When the Model Doesn't Know the Word: Lexical Coverage as the Hidden Driver of Distraction

Language models are easily thrown off by irrelevant text in their prompt, and they also stumble on words coined after their training data was collected. These two weaknesses are usually studied separately. We argue they compound in a specific, predictable way. To ignore a distracting passage, a model must first judge which passage is actually about the question. That judgment depends on understanding the key word in the question. When the key word is new, such as a neologism, fresh slang, or a familiar word with a new sense, the model loses its main tool for telling signal from noise. It then drifts toward passages that use words it already knows.

This matters because retrieval is the standard fix for words a model has never seen. The fix has only been validated in a sterile setting. NewTerm, a benchmark built from annual dictionary updates, shows two things. First, new terms cost models over 20% accuracy. Second, prepending one clean dictionary definition recovers "over 25 points on average." Real retrieval does not return a clean definition. It returns web pages where the definition sits among chatter, lookalike words and outdated senses of the term. We propose to measure how much of that 25-point gain survives realistic noise, to test whether the loss is larger for new terms than for familiar ones, and to test a simple remedy. The remedy is to have the model write a short definition of the term from the retrieved text before it answers, and to do this only when the term is rare in pretraining data.

Core Claim

Our central hypothesis is that distraction is moderated by lexical coverage, meaning how well the model's training data covered the words in the query. Adding noisy passages should hurt more when the query's key term is new than when an established term expresses the same content. The largest damage should come from one kind of distractor: a passage that uses the term, or a similar-looking familiar word, in an older or more common sense.

This follows from two findings in the file. NewTerm catalogs "Preference for the Familiar" (substituting a known word or sense) and "Ignoring" (guessing from surrounding context) as the dominant failure modes on new terms. Thinking in a Crowd shows that reasoning models build elaborate justifications around misleading context, a behavior the authors call the "Rationalization Trap." We predict the combination: a model that doesn't know a word will rationalize the familiar-sense distractor.

To quantify this we define gold retention. Let $A_{\text{base}}$ be accuracy with no context, $A_{\text{gold}}$ accuracy with the clean definition, and $A_{\text{noisy}}$ accuracy with the definition embedded among $k$ distractor passages. Then

$$ R(k) = \frac{A_{\text{noisy}}(k) - A_{\text{base}}}{A_{\text{gold}} - A_{\text{base}}}, $$

which is the fraction of the clean-definition benefit that survives noise. The claim is that $R_{\text{new}}(k) - R_{\text{established}}(k)$ is negative and grows in magnitude with $k$. The prediction is also that the gap is concentrated in the familiar-sense distractor condition.

Relation to Prior Work

Two papers motivate the question without answering it. NewTerm supplies the terms and the clean-context ceiling but never adds noise. Measuring the Impact of Lexical Training Data Coverage on Hallucination Detection builds a suffix-array index (a data structure for fast exact-string lookup) over the 1.3-trillion-token RedPajama corpus. With those counts, it raises hallucination detection on NQ-Open from 64.2% to 73.3% accuracy. It works only in closed-book settings, with no retrieved context at all. We reuse its index as a measurement tool for coverage, not as a detector.

Three training-free or trained defenses against distraction constrain our design, and all three were validated only on standard-vocabulary QA.

  • DAS. Distraction-aware answer selection adds an abstention instruction and re-scores each answer by $P(\text{query}\mid\text{document})$, the probability that the passage would generate the query. It reports a 64% average exact-match gain over naive zero-shot readers. That score depends on the model's likelihood of the query words, which may fail when those words are new. It may also systematically favor passages that use the familiar sense.
  • Ext2Gen. Ext2Gen trains a generator with preference optimization to extract evidence sentences before answering. It reports 0.557 against 0.473 for default Llama3.1-8b-instruct and beats standalone compressors. It extracts sentences, not senses.
  • SKR. SKR decides whether to retrieve at all using nearest neighbors over sentence embeddings. It also found that directly asking the model whether it knows something was the weakest option. We borrow its lesson that a gate should use external signals rather than self-report.

Our method's core step is borrowed from definition generation for semantic change. That work writes a definition of a word from each context it appears in, then compares the definitions, and this beats raw token embeddings for detecting sense shift (Spearman 0.637 vs 0.514 on SemEval-2020 English). It also finds greedy decoding cleaner than beam search, which we adopt. We apply the same "definitions as senses" idea not to track change over time, but to pick the right sense out of noisy retrieved passages.

Research Program

The work has three parts.

A controlled noise benchmark built on NewTerm. For each NewTerm item (the three task types, called COMA, COST and CSJ), we:

  • embed the gold definition in a realistic web-style passage;
  • add $k \in \{0, 2, 5, 10, 25\}$ distractors, following Ext2Gen's upper limit of 25 chunks.

The distractors come in three types: unrelated passages, passages about orthographically or semantically similar established words, and, for semantic-shift terms, passages using the older sense. Each item gets a matched control. We rewrite it so that an established term carries the same meaning, keeping the task and distractors fixed. This matching is what lets us attribute any gap to lexical coverage rather than to item difficulty.

We add a realistic condition in which contexts come from BM25 and dense retrieval over a web snapshot, with no planting. This reflects the concern in the file that dense retrievers handle unfamiliar vocabulary poorly. We also port the familiar-sense distractor idea to SciAux, so the Rationalization Trap finding can be checked on lexically novel versus familiar premises.

Coverage-gated definition distillation. The gate first computes the pretraining frequency of the key term's n-grams from the RedPajama index. If the term falls below a coverage threshold tuned on a development split, the pipeline:

  1. has the model write a one-sentence definition of the term from each retrieved passage separately, using greedy decoding;
  2. embeds the definitions and clusters them;
  3. discards definitions that echo the model's own no-context definition of the term, because those likely reflect the familiar sense;
  4. answers using only the chosen definition, which approximates NewTerm's Gold condition.

Above the threshold, the model reads the context normally. The gate exists because distilling a definition should not help, and may hurt, when the model already knows the word.

Analysis. We will tag errors as Ignoring, Preference for the Familiar, or Literalism, and measure how often each occurs as $k$ grows. For thinking-mode models, we will measure the length of reasoning traces on familiar-sense distractors.

We will use RedPajama-INCITE 3B and 7B, where coverage counts are exact, and Llama3.1-8b-instruct and Qwen3-8B with thinking on and off, where RedPajama counts serve only as a proxy. This proxy is a stated limitation. Our secondary gate, described in the ablations below, tests whether it matters.

Baselines

Naive reader. The model reads all passages with a plain prompt. This is the reference point for $R(k)$.

Oracle and floor. The Gold condition, with only the clean definition, gives the ceiling. The Base condition, with no context, gives the floor. The file does not report NewTerm accuracy under noisy contexts, only the clean-definition gain of over 25 points.

DAS. This is the strongest training-free competitor. It needs no training data, applies to any model, and its query-likelihood score directly targets distraction. We run it with the same passages and the same backbones, and give it the same development split to tune its score weighting. The file reports only the 64% average exact-match gain, measured on NQ, TriviaQA, WebQuestions and SQuAD with FLAN-T5-XL and OPT-IML-MAX. It reports no result on new-term inputs.

Ext2Gen. This is the strongest trained competitor, and it is the closest method in spirit because it isolates evidence before answering. If the released checkpoint is unavailable, we reproduce its preference-training recipe on Llama3.1-8b-instruct using its own synthetic-noise data. We deliberately use none of our NewTerm-derived data, so it is not disadvantaged relative to our method, which is also training-free on the benchmark. Its reported reference is an average score of 0.557 against 0.473 for the default model.

What counts as beating these baselines. Our method must show higher $R(10)$ and $R(25)$ than both DAS and Ext2Gen on new terms. It must also stay within one point of the naive reader on established-term controls. Otherwise it is only shifting errors around.

Ablations. Three ablations isolate the key ingredients:

  • Always-on distillation, with no gate, tests whether gating matters.
  • A log-probability gate and an SKR-style nearest-neighbor gate replace the corpus-count gate. They test whether coverage counts are the useful signal or whether any uncertainty signal works.
  • Distillation without the familiar-sense filter tests whether explicitly separating senses matters, or whether writing definitions is enough.

Why not just…?

Why not just use Ext2Gen, which already extracts evidence under noise? This is the most serious threat. Ext2Gen may simply extract the defining sentence and succeed. Our concern is the familiar-sense distractor. Sentence extraction trained on standard QA has no reason to prefer an unfamiliar sense over a fluent passage about a familiar one. Ext2Gen's own comparison did not include this case. The benchmark settles it directly. If Ext2Gen matches our method on familiar-sense distractors for new terms, the distillation step adds nothing and we will report that.

Why not just use DAS's query-likelihood rescoring? It is cheap and strong. However, $P(\text{query}\mid\text{document})$ is computed by the same model that has never seen the term. It may assign higher likelihood to passages that use the term in its familiar sense. The coverage paper likewise shows that likelihood signals are unreliable on unrepresented spans. We will directly check whether DAS scores rank familiar-sense distractors above true definitions for new terms but not for controls.

Why not just tell the model the word may be new and to trust the context? SKR found that the model's self-assessment of its own knowledge is unreliable. But an instruction is not self-assessment, so this remains an open question. We include the instruction prompt as a baseline. If it recovers most of the lost retention, the practical recommendation becomes a prompt change and our pipeline is unnecessary.

Why not just use a reasoning model or a bigger model? NewTerm reports that large models such as Llama-3-70B and GPT-4 degrade on new terms much as small models do. Thinking in a Crowd reports that thinking mode made Qwen3-8B fall from 96.22% to 77.03% under misleading snippets. We still run Qwen3-8B with thinking on and off. A reasoning model that escapes familiar-sense distractors would contradict our hypothesis and would be reported as such.

Decisive Checks

The first check is the interaction itself: is $R_{\text{new}} - R_{\text{established}}$ significantly negative, with the largest effect under familiar-sense distractors? If the gap is near zero, lexical coverage does not moderate distraction. The benchmark and that null result then become the contribution, and the method is dropped.

The second check is whether gated distillation beats DAS and Ext2Gen on new-term retention at $k \ge 10$ without hurting established-term controls. This must hold in the realistic-retrieval condition as well as the planted condition.

The third check is whether the corpus-count gate beats the log-probability and nearest-neighbor gates. If it does, external coverage statistics carry information the model's own confidence lacks. That would extend the closed-book finding of the coverage paper to retrieval with noisy context.