Deep neural networks are remarkably good at learning smooth, generalizable representations of data—a property often described as "eager learning." However, they struggle with "long-tail" memorization, rapid domain adaptation, and dynamic context updates. To update what a model knows, traditional approaches require expensive fine-tuning or backpropagation.
An elegant alternative bypasses weight updates entirely by separating a system's capabilities into two distinct components: a parametric neural network that provides general linguistic or structural priors, and a non-parametric datastore that holds explicit, localized facts. Instead of forcing the network to digest retrieved text through its input context window, this approach performs late fusion at the final step of inference. It interpolates the model's output probability distribution with a distribution derived directly from retrieved neighbors.
The Mechanics of Distribution Interpolation
How does this work under the hood? Consider an autoregressive language model predicting the next token. At any given step, the model maps the current context to a dense hidden vector $h$.
In a standard setup, this vector is multiplied by the output embedding matrix to produce a vocabulary-wide probability distribution, $P_{LM}$. In an interpolated retrieval-augmented setup, such as the $k$NN-LM framework popularized by Urvashi Khandelwal et al. in Nearest Neighbor Machine Translation, the vector $h$ also serves as a query into an external datastore.
This datastore contains pairs of cached hidden states and their corresponding ground-truth target tokens from a reference corpus. The retrieval mechanism identifies the $k$-nearest neighbors to $h$ based on a distance metric (such as negative $L_2$ distance or dot product). These neighbors are then converted into a sparse probability distribution over the vocabulary, $P_{kNN}$, where the probability of a token is proportional to the exponential of its negative distance to the query.
The final step is a simple linear interpolation of the two distributions:
$$P_{final}(w) = (1 - \lambda) P_{LM}(w) + \lambda P_{kNN}(w)$$
Here, $\lambda \in [0, 1]$ is a tunable hyperparameter that acts as a knob, balancing the model's internal parametric knowledge against the explicit evidence found in the external datastore.
Why It Works: Bypassing the Representation Bottleneck
To understand why late-stage interpolation is so effective, it helps to contrast it with standard Retrieval-Augmented Generation (RAG), where retrieved text is prepended directly to the model's prompt.
When out-of-domain or highly technical text is forced into a model's context window, the model's self-attention layers must process unfamiliar token combinations. If the domain shift is severe, the internal representations can degrade, leading to hallucinations. The authors of SILO Language Models observed that interpolating token probabilities directly into the final output distribution is significantly more robust to domain shifts than feeding retrieved text into the context. This is because the model's hidden space remains highly capable of measuring similarity (finding the right neighbors) even when it lacks the internal parametric capacity to generate or reason about that domain's specific linguistic patterns from scratch.
Furthermore, this mechanism provides a robust safety net. As shown by Lei Li et al. in Revisiting k-NN for Fine-tuning Pre-trained Language Models, combining a model's predictions with training-set neighbors acts as an effective regularizer, boosting performance dramatically in data-scarce zero-shot and few-shot scenarios by leveraging explicit data retrieval when parametric knowledge is insufficient.
Architectural Variations and Extensions
While linear interpolation via a weighted sum is the most common approach, researchers have developed alternative mathematical formulations to handle complex scenarios:
- Continuous Caching: Early foundations laid by Edouard Grave et al. in Improving Neural Language Models with a Continuous Cache used a local context cache to dynamically interpolate long-term vocabulary statistics with immediate local history, allowing recurrent models to adapt to recent context smoothly.
- Product of Experts: Instead of a linear sum, Luiza Amador Pozzobon et al. in Goodtriever leverage a product-of-experts approach to combine distributions. This allows the retrieval mechanism to strictly suppress or boost specific toxic tokens in real time based on external evidence.
- Power Interpolation: To resolve acute knowledge conflicts between a model's internal memory and external context, Anant Khandelwal et al. introduced a confidence-aware power interpolation method in CoCoA, ensuring a smoother, more controlled integration of conflicting information.
Ubiquitous Utility Across Modalities and Tasks
Because distribution interpolation is a "plug-and-play" inference-time modification, it has been successfully adapted across a wide array of domains without requiring core model retraining:
- Speech and Translation: Yichao Du et al. utilized weighted interpolation to achieve massive domain adaptation gains in speech translation. Similarly, Maya K. Nachesa applied $k$NN interpolation to OpenAI's Whisper model to enhance speaker adaptation and mitigate bias, while Puneet Mathur et al. used domain-distributed co-occurrence matrices in DOC-RAG to bias speech recognition toward user-specific jargon.
- Structured Extraction and Captioning: In token-level classification tasks like Named Entity Recognition and skill extraction, Ahmed Abdou & Tasneem Mahmoud as well as Mike Zhang in NNOSE demonstrated that interpolating $k$NN distributions helps capture fine-grained, long-tail entities. For multi-modal tasks, Zhengcong Fei showed that linearly interpolating a standard image captioning model's predictions with a non-parametric memory bank yields state-of-the-art results without adding new training parameters.
Key Foundational Work
Incorporating Discrete Translation Lexicons into Neural Machine Translation (2016) — contributed 1 concept elements to this direction.
Neural Network Language Model for Chinese Pinyin Input Method Engine (2015) — contributed 1 concept elements to this direction.