Standard large language models (LLMs) are "parametric" — they store everything they know within the fixed weights of their neural network. While powerful, this approach forces the model to compress the entire internet into a finite set of parameters, often leading to struggles with rare facts, domain-specific jargon, and rapid updates.
A growing body of research, pioneered by Urvashi Khandelwal, Mike Lewis, and Luke Zettlemoyer, suggests a different path: $k$-Nearest Neighbor Language Models ($k$NN-LMs). Instead of relying solely on internal weights, these models "look up" similar examples from a massive external datastore during inference to refine their predictions.
The Mechanism of Explicit Memory
The core intuition behind a $k$NN-LM, as detailed in Generalization through Memorization: Nearest Neighbor Language Models, is to treat the training data as an explicit, searchable memory.
The process works in three steps:
- Datastore Construction: The model processes a large corpus and saves the "hidden state" (the mathematical representation of the context) for every single token as a key, and the actual next token as the value.
- Retrieval: At test time, when the model is predicting the next word, it uses its current hidden state to query the datastore for the $k$ most similar keys.
- Interpolation: This is the critical step. The model doesn't just pick the most common retrieved token. Instead, it calculates a probability distribution from these neighbors and linearly interpolates it with its own internal softmax distribution.
The final prediction is a weighted average: $P = \lambda P_{kNN} + (1 - \lambda) P_{LM}$. By tuning the interpolation parameter $\lambda$, researchers like Frank F. Xu and Graham Neubig have shown that models can balance the "smooth" general linguistic patterns of the neural network with the "sharp" specific evidence found in the datastore.
Solving the Long Tail and Domain Shift
The primary advantage of this approach is its ability to handle the "long tail" of data — rare tokens that appear too infrequently for a parametric model to memorize reliably. For instance, Taku Sakamoto and Akiko Aizawa demonstrated that $k$NN-LMs significantly improve the prediction of numerals, which standard models often treat as arbitrary strings rather than magnitudes. Similarly, Tong Ye and Lingfei Wu applied this to source code summarization, where retrieving specific technical terms from a datastore (the Tram mechanism) proved more effective than relying on general weights.
Beyond rare tokens, interpolation allows for instant domain adaptation. In Nearest Neighbor Machine Translation, Urvashi Khandelwal and Angela Fan showed that a general-purpose translation model could become a "legal expert" simply by swapping its datastore for one containing legal documents, requiring zero additional training.
Isolating Risk and Ensuring Privacy
A recent and vital application of this architecture is the SILO Language Model, proposed by Sewon Min, Suchin Gururangan, and Eric Wallace.
In the face of copyright and privacy concerns, SILO trains its parametric weights only on "low-risk," permissive data. It then uses a non-parametric datastore for "high-risk" or copyrighted data. This allows for:
- Data Attribution: The model can point exactly to the retrieved passage that influenced its answer.
- The Right to Erasure: If a user or creator wants their data removed, the developer can simply delete those entries from the datastore without having to retrain the entire model.
The Perplexity Paradox
While $k$NN-LMs are famous for drastically reducing perplexity (the mathematical measure of how well a model predicts the next token), recent work has highlighted a "perplexity paradox."
Shufan Wang and Mohit Iyyer found that while $k$NN-LMs excel at predicting the next token in a static test set, they do not necessarily improve open-ended text generation. The gains are often concentrated in a small subset of tokens, and the model's overall coherence doesn't always scale with its lower perplexity.
Furthermore, Alexander M Rush and Shangyi Geng argue in Great Memory, Shallow Reasoning: Limits of kNN-LMs that while these models have an incredible capacity for memorization, the interpolation mechanism can sometimes degrade a model's ability to perform multi-hop reasoning or complex deduction. The model becomes so reliant on "looking up" the answer that it fails to "think" through the logic.
Why It Works: Ensembling and Regularization
If the gains aren't just from rote memorization, why do they work so well? Frank F. Xu suggests that $k$NN-LMs act as a form of representation ensembling. By looking at the hidden states of many different training examples, the model effectively averages out the noise in its own internal representations. Surprisingly, even using approximate nearest neighbor search acts as a form of regularization, preventing the model from over-fitting to specific, idiosyncratic examples in the training set.