Retrieval-augmented language models of the nearest-neighbor kind attach a large memory of (context vector, next token) pairs to a frozen model. At every decoding step they look up the stored contexts closest to the current one and mix the tokens that followed them into the prediction. This helps with rare domain facts. It also adds a slow, memory-bound vector search to every generated token. The methods that skip unnecessary searches all train a separate gating network on labeled data. We propose to test a simpler possibility. If a domain model is adapted with low-rank adapters (LoRA: small trainable matrices added to frozen weights), its own predictive entropy may remain an honest estimate of how wrong it is about to be. If so, the model can decide when to search with one scalar threshold, and that threshold can be derived from the datastore text alone, with no gate network and no labeled tuning set.
The two-sentence version is this. Full fine-tuning tends to make models overconfident, but LoRA appears to preserve the calibration learned in pre-training. Our hypothesis is that a LoRA-adapted, domain-pretrained model can therefore gate its own retrieval, searching only when its entropy is high, and match trained skip-classifiers on the speed/quality trade-off while needing no labels. The proposal also delivers a label-free diagnostic, computed on raw datastore text, that predicts in advance whether this self-gating will work for a given model.
Core Claim
Write $p_t$ for the model's next-token distribution at step $t$, $H(p_t)$ for its entropy, and $y_t$ for the token that actually follows. If the model is calibrated, meaning its predicted distribution matches how often tokens actually occur given its own predictions, then in expectation
$$\mathbb{E}\left[-\log p_t(y_t)\right] = H(p_t),$$
because entropy is exactly the expected log-loss under the model's own distribution. Retrieval can only lower loss where loss is high. A calibrated model's entropy is therefore a direct, free proxy for how much a datastore lookup could help.
Our rule is to search only when $H(p_t) > \tau$. The threshold $\tau$ is chosen on a held-out slice of the datastore's own text, where next tokens are available without annotation. We run leave-one-out retrieval on that slice, measure the average loss reduction from interpolating with retrieved tokens in each entropy bin, and set $\tau$ at the entropy where that reduction falls below a stated exchange rate between nats saved and searches spent. The exchange rate is the only user-chosen quantity, and it expresses a latency budget rather than a hyperparameter search.
The same text yields the diagnostic: the gap between average entropy and average realized loss, a calibration error that needs no labels. Our prediction is directional. LoRA-adapted models will show a small gap and gate well. Fully fine-tuned models will show entropy below realized loss, skip searches they needed, and lose quality.
Relation to Prior Work
The work that motivates the gap trains gates under supervision. Efficient k-Nearest-Neighbor Machine Translation with Dynamic Retrieval trains a two-layer classifier on top-1 probability, hidden-state norm and cross-attention, combined with a timestep-decaying threshold. It reports that raw model confidence is too unreliable to gate with directly. Adaptive Nearest Neighbor Machine Translation trains a tiny network that picks the neighbor count, including zero, from neighbor distances. That network needs the search to have run already, so it cannot save search cost. You can't pick your neighbors, or can you? makes the mixing weight depend on retrieval distance, tuned per distance bucket on validation data. It still searches at every step.
We differ from all three in the same way. We ask whether a property of the adaptation method makes the trained gate unnecessary, and we take that property as the object of study. We also attack head-on the dynamic-retrieval paper's finding that confidence is unreliable: our hypothesis is that this finding reflects miscalibration introduced by how the model was trained, not something inherent to confidence.
The mechanism comes from On the Calibration of Large Language Models and Alignment. That paper reports expected calibration error rising from near zero to above 0.20 on MMLU after full instruction tuning, while LoRA preserved pre-trained calibration. It gives no exact number for LoRA, and its evidence comes from instruction tuning rather than domain adaptation. Whether the effect transfers is the first thing we test.
Two results constrain the design. First, Cruz et al. find that large modern networks produce larger feature norms on unfamiliar inputs. This makes the raw hidden-state norm used by the dynamic-retrieval classifier a signal whose meaning may flip between architectures, so we avoid unnormalized norms. Second, Cruz et al. and Xu & Klabjan set thresholds by fitting extreme-value distributions to score tails. We borrow this only for a secondary purpose: re-estimating the search rate on unlabeled deployment traffic when it drifts away from the datastore text.
The application regime comes from domain pre-training. RadAdapt and Männistö et al. show that the pre-training corpus matters more than the adaptation method, and that LoRA comes close to full fine-tuning. These results make "domain-pretrained base plus LoRA" the realistic deployment point. Neither paper attaches a datastore.
Research Program
We will build a single gating rule and measure it across three regimes. Each regime varies one factor the hypothesis depends on.
The first regime is a general-domain sanity check: the WikiText-103 nearest-neighbor language model setting of Khandelwal et al., with the base model left as is and also LoRA-adapted on the datastore text.
The second is multi-domain German–English translation (IT, Medical, Law, Koran, Subtitles). This is where the supervised skip-classifiers report numbers, so it allows a direct comparison. Here the general translation model is adapted to each domain either by LoRA or by full fine-tuning on part of the domain data, and the datastore is built from all of it.
The third is the setting the proposal is about: radiology report summarization on MIMIC-III. We use T5 and Clinical-T5 bases, adapted with LoRA through the davevanveen/radadapt pipeline, and add a token-level datastore built from the training findings–impression pairs. This regime crosses the pre-training axis (general versus clinical) with the adaptation axis (LoRA versus full fine-tuning).
For every model we compute the label-free calibration gap on held-out datastore text before any downstream evaluation. We then check whether that gap predicts how much quality the entropy gate loses relative to always searching. This is the analysis that turns a heuristic into something one can trust before deployment.
Baselines
The naive reference points are no retrieval and always retrieve. For WikiText-103, You can't pick your neighbors reports perplexity 18.65 for the base model and 16.12 for vanilla nearest-neighbor augmentation. For radiology, RadAdapt reports 16.2 BLEU and 40.8 F1-RadGraph for LoRA-adapted Clinical-T5 without retrieval. Every gated method lies on a curve between these two endpoints, and we report full quality-versus-search-fraction curves rather than single points.
The strongest existing competitor is the supervised dynamic-retrieval classifier of Gao et al.. It is the only published method that actually removes searches, and it reports 41.83 average BLEU (0.74 below vanilla) with Subtitles decoding raised from 478 to 1002 tokens per second at batch size 128. We will rerun it on our own adapted models rather than copying numbers. It gets the same datastores, the same FAISS index, the same hardware and batch size, and its labels drawn from the same domain training text our threshold uses. Its tuning budget will match ours in wall-clock time. Beating it means reaching equal or better BLEU at equal or higher throughput on at least three of the five domains, with no trained gate. Matching it within noise would still be a meaningful result, because ours needs no labels or training.
The adaptive neighbor-count network of Zheng et al. and the distance-bucketed mixing weight of Drozdov et al. are run as quality references. Both search every step, so they bound attainable quality rather than speed. The compressed-key method of Wang et al. is orthogonal; we stack our gate on top of it to show the savings compound.
Three ablations isolate the key ingredient. The first applies the identical entropy rule to fully fine-tuned models; if they gate as well as LoRA models, the calibration story is wrong. The second replaces entropy with the raw hidden-state norm, testing the sign-flip concern from Cruz et al. across general and clinical bases. The third replaces our datastore-derived threshold with a fixed confidence cutoff.
Why not just…?
Why not just threshold top-1 probability at a fixed value like 0.8? This is the most serious threat, because our rule is itself a threshold on the model's own confidence. Our contribution therefore depends on two claims. The first is that a fixed cutoff lands at very different points on the trade-off curve across domains and bases. The second is that our datastore-derived threshold lands near the best point on each curve without per-setting tuning. The fixed-cutoff ablation settles this directly. If one constant works across all settings, the threshold derivation adds nothing, and the remaining contribution is the narrower calibration finding about LoRA versus full fine-tuning. Prior evidence is mixed: Gao et al. report poor trade-offs for static thresholds, but they did not test LoRA-adapted models.
Why not just train a small gate, since Meta-k reportedly works with as few as 100 sentences? Zheng et al.'s network needs the retrieval to have happened already, so it cannot reduce search cost. Gao et al.'s gate does reduce cost but needs labels and a training run for each model and domain. A cheap supervised gate is still a fair competitor. If it beats the entropy rule by a clear margin at equal labeling cost, the right conclusion is that calibration helps but is not enough, and we will report it that way.
Why not just cache lookups, as PROXIMITY does? Bergman et al. cut retrieval latency by 72.5% on skewed retrieval-augmented generation workloads. That gain depends on queries repeating. Token-level context vectors change at every step, so we expect a low hit rate. We will measure that hit rate on our decoding traces. In any case caching is complementary to gating and can be layered on top.
Why not drop the datastore and just fine-tune the domain model? For the clinical setting, the no-retrieval endpoint is exactly this baseline. If always-retrieve does not beat LoRA-adapted Clinical-T5 on F1-RadGraph, the latency question is moot there, and we will say so. Khandelwal et al.'s finding that a datastore beats training on 30 times more data suggests retrieval will help with rare findings, but this needs to be shown in the radiology setting specifically.
Decisive Checks
Three results would settle the proposal.
The first is about calibration. On held-out datastore text, the label-free calibration gap should be clearly smaller for LoRA than for full fine-tuning in all three regimes. If it is not, the mechanism fails and we stop.
The second is about whether the diagnostic is useful. Across all models, that gap should rank-correlate with the quality the entropy gate loses relative to always searching.
The third is about the head-to-head comparison. On the translation domains, the entropy gate on LoRA-adapted models should reach within Gao et al.'s reported 0.74 BLEU of vanilla retrieval while at least doubling throughput, and on clinical summarization it should preserve most of the F1-RadGraph gain from retrieval while skipping at least half the searches.
If the first check passes but the third fails against the fixed-cutoff ablation, the paper becomes a calibration study rather than a gating method, and we will present it that way.
References
- Efficient k-Nearest-Neighbor Machine Translation with Dynamic Retrieval
- Adaptive Nearest Neighbor Machine Translation
- You can't pick your neighbors, or can you? When and How to Rely on Retrieval in the kNN-LM
- On the Calibration of Large Language Models and Alignment
- Operational Open-Set Recognition and PostMax Refinement
- Open Set Domain Adaptation by Extreme Value Theory
- RadAdapt: Radiology Report Summarization via Lightweight Domain Adaptation of Large Language Models
- A Comparative Study of PEFT Methods for Python Code Generation
- Generalization through Memorization: Nearest Neighbor Language Models
- Efficient Cluster-Based k-Nearest-Neighbor Machine Translation
- Leveraging Approximate Caching for Faster Retrieval-Augmented Generation