Beyond the Black Box: Implementing KNN for Feature Similarity from Scratch

As Web101 by Han expands into deeper technical systems, it is not enough to just call an API. To build truly resilient AI systems, we need to understand the underlying geometry of decision-making. Today, we are looking at K-Nearest Neighbors (KNN), a fundamental classification algorithm, and building it from first principles to see how similarity is actually calculated in a multi-dimensional space.

At its core, KNN assumes that similar data points exist in close proximity. If you have a messy dataset of web performance metrics, KNN can classify a new instance, such as a page load event, by looking at its k closest neighbors in feature space. It is a lazy learner because it does not build a model during training. Instead, it does the heavy lifting during prediction time.

To find the nearest neighbor, we need a way to measure distance. The most common method is Euclidean distance. If we compare two points, x = (x1, x2) and y = (y1, y2), in a 2D space, the formula is: ```text d(x, y) = sqrt((x1 - y1)^2 + (x2 - y2)^2) ``` In a machine learning context, these values correspond to your features, such as page weight, script execution time, and server response latency. KNN uses that distance to decide which stored examples are most similar to the new input.

Here is a raw Python implementation. By avoiding high-level libraries like Scikit-Learn for this demonstration, we can see the exact loop that determines similarity. ```python import numpy as np from collections import Counter def euclidean_distance(x1, x2): return np.sqrt(np.sum((x1 - x2)**2)) class KNN: def __init__(self, k=3): self.k = k def fit(self, X, y): self.X_train = X self.y_train = y def predict(self, X): predictions = [self._predict(x) for x in X] return predictions def _predict(self, x): # 1. Compute the distance to all training points distances = [euclidean_distance(x, x_train) for x_train in self.X_train] # 2. Get the indices of the k nearest neighbors k_indices = np.argsort(distances)[:self.k] # 3. Extract the labels of those k neighbors k_nearest_labels = [self.y_train[i] for i in k_indices] # 4. Return the most common class label (majority vote) most_common = Counter(k_nearest_labels).most_common() return most_common[0][0] ``` This makes the full prediction path explicit: compute all distances, sort them, select the nearest labels, and return the majority class.

One major challenge in machine learning systems is the curse of dimensionality. As you add more features, the distance between points becomes less meaningful because everything starts to look far away. That weakens the usefulness of nearest-neighbor comparisons and can reduce classification quality.

KNN is highly sensitive to feature scale, so you should normalize or standardize your data before running it. If one feature is page weight in bytes and another is load time in seconds, the byte-scale feature can dominate the distance calculation and drown out the smaller feature. Proper scaling prevents that imbalance and makes similarity comparisons more meaningful.

Understanding KNN is a first step toward mastering vector search and other similarity-based systems. By deconstructing these black-box algorithms, we gain the ability to troubleshoot, reason about, and optimize technical systems at the architectural level instead of treating them like magic.