Don't Skip Retrieval When the Model Is Confidently Dull

Retrieval-augmented models keep facts in an external store and look them up while generating text. Each lookup is slow, so efficient systems train a small classifier that skips the lookup whenever the model already seems sure of its next word. In open-ended conversation this rule has a blind spot. A dialogue model is most sure of itself when it is about to say something empty, such as "I see" or "that sounds fun." Those generic replies are what a model produces when its weights do not hold the relevant fact. So a confidence-based skipping rule turns off retrieval at the moment the conversation most needs outside knowledge.

This note proposes to measure that failure directly and to fix it with a different skipping rule. The proposed rule is trained on the model's own generated text rather than on human reference text. Its training signal asks whether retrieving at this point would change the reply into one that uses grounded, copyable content. It also reads the model's copy gate, the part of a pointer-generator network that decides whether to generate a word or copy it from retrieved text. The goal is the speed of adaptive retrieval without the dullness that such skipping quietly brings back.

Core Claim

Skipping classifiers of the kind used in Efficient Nearest Neighbor Language Models and Efficient k-Nearest-Neighbor Machine Translation with Dynamic Retrieval make two assumptions. They are trained on human-written prefixes, and they treat the model's confidence as a sign that retrieval is not needed. At inference, a dialogue model follows its own prefixes, and these drift toward generic replies that the classifier never saw during training. On such prefixes the model's high confidence means that its answer does not depend on the context, not that it knows the answer.

We claim three things. First, confidence-based skipping raises the rate of generic replies relative to always retrieving, by more than it lowers any quality metric that is tied to likelihood. Second, a skipping policy trained on self-generated prefixes, with labels based on whether retrieval changes the outcome, recovers most of the informativeness of always retrieving. Third, it keeps a large share of the speedup. The new element is not a copy mechanism and not adaptive retrieval as such, since both exist. It is the choice of training distribution and label for the skipping decision, together with a diagnostic that exposes the failure.

Where This Sits

Three bodies of work motivate the claim.

The first concerns capacity. Transformers store at most about two bits of fact per parameter (Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws), and nearing that ceiling takes very long training (Scaling Laws for Fact Memorization of Large Language Models). This is why long-tail facts end up in external stores such as RETRO's trillion-token database (Improving Language Models by Retrieving from Trillions of Tokens). Our contribution is to note that each skipped retrieval hands the decision back to these limited parameters.

The second concerns generic replies. Models trained to maximize likelihood fall back on safe, context-independent responses (Overview of the NTCIR-13 Short Text Conversation Task; The Second Conversational Intelligence Challenge (ConvAI2)). Grounding helps. Wizard of Wikipedia retrieves Wikipedia passages and lifts human engagingness from 2.11 to 2.92 out of 5. DeepCopy points hierarchically into persona facts and roughly doubles Distinct-n, a count of unique word n-grams used as a diversity measure.

The third concerns speed. Adaptive retrieval shows that most lookups are wasted: nearest-neighbour retrieval helps only 39% of the time on WikiText-103, per Efficient Nearest Neighbor Language Models. It also shows that the usefulness of retrieval decays over the course of decoding (Efficient k-Nearest-Neighbor Machine Translation with Dynamic Retrieval).

Some work supplies the method. LargePiG turns a decoder language model into a pointer-generator without training. It uses attention weights as the copy distribution and sets the copy probability from the divergence between the model's intermediate and final layers, at about 6% extra latency on LLaMA2-7B. We reuse its copy distribution over retrieved text, and we reuse its copy probability as a cheap feature for the skipping decision. A Copy-Augmented Generative Model for Open-Domain Question Answering adds 1,537 parameters to Fusion-in-Decoder, a reader that encodes each retrieved passage separately and merges them in the decoder. That addition raises NaturalQuestions exact match from 48.5 to 51.4 with 25 passages instead of 100. This shows that copying makes retrieved text more useful per unit of cost, which matters when every lookup is budgeted. Subset Retrieval Nearest Neighbor Machine Translation supplies the caching idea: one coarse lookup restricts later fine-grained search.

Some work constrains the design. DeepCopy reports that humans use persona facts only about 49% of the time. ConvAI2 found that an over-grounded model asked 107 questions per 100 dialogues, against 12 for humans. A policy that simply always retrieves and copies is therefore not the goal, because the policy must also learn when not to ground.

None of these papers trains the skipping decision on the model's own trajectories, and none evaluates skipping against how generic the responses become.

Research Program

Diagnostic study. The first piece is a diagnostic that exposes the failure. We need an operational definition of a generic reply. We call a model reply generic when nearly the same string (above a fixed token-overlap threshold) appears among the model's replies to many unrelated dialogue contexts in a held-out pool. This is a direct, testable form of context-independence.

We then train a He-et-al.-style skipping classifier on gold prefixes from Wizard of Wikipedia and PersonaChat. We record its skip rate at the first tokens of generic versus specific model replies. The prediction is that the skip rate is markedly higher on generic onsets. If it is not, the premise fails and we report that as a negative result.

Proposed system. The second piece is the system. A decoder model uses LargePiG-style copying over a small per-turn retrieval cache. A retrieval event fetches passages from a Wikipedia index for Wizard of Wikipedia, or a pooled persona-fact index for PersonaChat. It puts them in the cache for the rest of the turn, so later tokens can copy from them without a new search. Cost is dominated by the number of retrieval events, which we count along with wall-clock tokens per second.

Counterfactual labels. The skipping policy is a small classifier trained on decoding states from the model's own greedy and sampled outputs. Its label compares the rollout after retrieving at step $t$ with the rollout after skipping:

$$y_t = \mathbb{1}\left[ S(\hat{r}_t^{\text{ret}}) - S(\hat{r}_t^{\text{skip}}) > \delta \right]$$

Here $\hat{r}_t^{\text{ret}}$ and $\hat{r}_t^{\text{skip}}$ are the reply continuations decoded after retrieving or skipping at step $t$. $S$ scores a reply by its unigram overlap with the gold knowledge sentence, minus a penalty if the reply is generic under the definition above. $\delta$ is a margin tuned on development data.

Computing $y_t$ needs two rollouts per sampled state. This is affordable offline because we label only a subsample of decision points.

Features. The classifier sees the usual confidence features: top-1 probability, hidden-state norm, and timestep. It also sees two features that ask whether retrieved text would be used. One is the copy probability that LargePiG's layer divergence assigns at this step. The other is the genericity score of the model's current partial reply.

Separating the ingredients. Two ablations isolate what matters. One changes only the training distribution, using confidence features on self-generated prefixes. The other changes only the label, using counterfactual labels on gold prefixes. This tells us whether the gain comes from the prefixes the classifier trains on, from the label, or from the copy-gate feature.

Baselines

Never retrieve. The naive reference point is the same decoder with no retrieval. It gives the speed ceiling and the genericity floor.

Always retrieve, then copy. The quality reference retrieves at the start of every turn and whenever the cache is refreshed, with LargePiG copying over the cache. This is the natural descendant of the Wizard of Wikipedia retrieval pipeline and of DeepCopy's grounded copying. Our policy must match its informativeness within a small margin, measured by knowledge-sentence overlap and human informativeness, while using clearly fewer retrieval events.

Confidence-based skipping (strongest existing competitor). This is the classifier of Efficient Nearest Neighbor Language Models, extended with the timestep-aware threshold and features of Efficient k-Nearest-Neighbor Machine Translation with Dynamic Retrieval. It is the strongest competitor because it is the published method for exactly this speed-versus-quality trade-off. It reports 277 to 1835 tokens per second (6.6x) on WikiText-103 at unchanged perplexity, and 1002 versus 478 tokens per second on Subtitles at a 0.74 BLEU cost. There is no reported result on Lacuna for either method on dialogue benchmarks, so we reimplement both.

To keep the comparison fair, all policies share the same base model, retrieval index, cache and hardware. Each skipping threshold is swept, and we compare full curves of informativeness against retrieval events and against tokens per second, not single operating points. Every classifier gets the same labelled-state budget and tuning budget.

Beating this baseline means one of two things. Either our curve dominates it, giving higher knowledge overlap and lower generic rate at matched tokens per second. Or it matches it on informativeness with a higher skip rate.

Turn-onset retrieval. One lookup at the first token of each turn, then no more. This is motivated by the finding that retrieval matters most early in decoding and by subset retrieval.

Decoding-only diversity. Nucleus sampling with a repetition penalty and no retrieval, tuned to match our Distinct-2.

Ablations. The two ablations described above, which change only the training prefixes or only the label.

Why not just…?

Why not just lower the confidence threshold? Lowering the threshold makes the classifier retrieve more everywhere. That buys informativeness only by giving up speed, and it does not target the generic onsets. The threshold sweep in the experiments tests this directly. If the confidence classifier's curve overlaps ours once the threshold is tuned, the failure is only a calibration problem, and we would report that rather than claim a new policy.

Why not just retrieve once at the start of every turn? This is the most serious threat. Generic replies are usually decided in the first few tokens, and usefulness decays over decoding, so one early lookup may capture most of the benefit. Two things count against it being sufficient. First, humans ground only about half their turns (DeepCopy), so retrieving on every turn wastes roughly half the lookups and pushes the model toward the over-grounded, question-heavy style seen in ConvAI2. Second, some replies turn toward a new fact partway through. If turn-onset retrieval matches our policy on the full curve, the honest conclusion is that dialogue needs a decision per turn, not per token, and our labelling scheme would be reduced to a turn-level classifier. That is still a useful result, and the experiment is designed to show it.

Why not just fix dullness with sampling? Sampling raises Distinct-n by picking rarer vocabulary, not by adding grounded content. DeepCopy's diversity gain came from copying concrete facts. Matching Distinct-2 against the sampling baseline and then comparing knowledge-sentence overlap and human informativeness shows whether the diversity carries information.

Why not just use a larger closed-book model? The two-bit-per-parameter ceiling (Physics of Language Models: Part 3.3) and the finding of "safe but useless" advice from modern models (Ask the experts) suggest that scale does not remove generic replies on long-tail topics. We test this on the Unseen split of Wizard of Wikipedia with a closed-book model several times larger, run at equal wall-clock cost.

Proposed Experiments and Decisive Checks

The main benchmarks are Wizard of Wikipedia, on both the Seen and Unseen topic splits, and ConvAI2/PersonaChat. For PersonaChat, persona facts are pooled into a retrieval index across all 1,155 personas rather than given to the model in its context. The NTCIR short-text conversation corpus, with its roughly 2 million retrieval candidates, is a secondary test at larger index scale.

The metrics are tokens per second, retrieval events per turn, generic-reply rate as defined above, Distinct-1 and Distinct-2, overlap with the gold knowledge sentence, perplexity, and human ratings of engagingness and informativeness. Reference points from the literature are Wizard of Wikipedia's 2.92 out of 5 engagingness for its knowledge model against 4.13 for humans, and DeepCopy's 3.67 out of 5 appropriateness on ConvAI2. These are context rather than direct targets, because base models differ.

First decisive check: does the failure exist? Confidence-based skipping must skip generic onsets at a clearly higher rate than specific onsets, and must raise the generic-reply rate relative to always retrieving. If neither holds, the proposal stops here.

Second decisive check: does the new policy beat the strongest competitor? It must dominate confidence skipping on the curve of informativeness against tokens per second, and not merely match it at one threshold.

Third decisive check: where does the gain come from? The two ablations must attribute the gain mainly to the change in training prefixes and labels, not to the copy-gate feature alone. If the copy-gate feature does all the work, the simpler story is that LargePiG's copy probability is a good retrieval trigger by itself, and we would report that story instead.

Final check: is the result real in human judgments? Human raters must confirm that lower generic rates do not come from over-grounded, question-heavy replies. We check this by counting questions per turn against the human rate in ConvAI2.