Retrieval-augmented generation (RAG) is a system that looks up passages in an external store and pastes them into a language model's prompt. Its most damaging failure is not missing evidence. It is plausible wrong evidence: a passage about a sister species or a neighbouring landmark that the model reads and then answers from with full confidence. This proposal asks whether a signal already present inside the model can flag these near-miss passages. The signal is the length (L2 norm) of the model's internal activations, measured with and without the passage. Face recognition learned years ago that the length of an embedding tracks how recognizable an input is, and recent work shows that hidden-state length in language models tracks moments of committed reasoning. We test whether the change in hidden-state length caused by adding a passage separates passages that actually answer the question from passages that are merely on-topic. If it does, the change can serve as a training-free score for choosing among passages and for abstaining.
The testbed is knowledge-based visual question answering, where near-misses are the dominant failure. In this task a model must name a fine-grained entity in an image and then look up facts about it. Two constraints shape the design. Output probabilities cannot be trusted as a relevance signal, because distracted readers put probability 0.98 on wrong answers (Cho et al.). The strongest existing fix scores passages by how well they regenerate the question. That measures topicality, and a sibling-entity passage is exactly the case where topicality and correctness come apart.
Core Claim
Our hypothesis concerns what happens when a supportive passage is added to the prompt. We expect the reader to commit to an answer, and the late-layer hidden-state norm to rise relative to the same question asked with no passage. We expect a near-miss passage to produce conflicting cues and a smaller rise, or none.
The hypothesis borrows from face recognition. There, degraded inputs pulled between class boundaries end up with shrunken embeddings (MagFace; AdaFace). It also borrows from the finding that late-layer norm spikes mark critical reasoning steps in language models (The Tell-Tale Norm).
The analogy has a real limit. Face-recognition norms are shaped by an explicit loss term, while language-model norms are not, so the direction of the effect has to be measured rather than assumed. We define the score as follows. Let $h_l(q,d)$ be the reader's hidden state at layer $l$, taken at the last prompt token and averaged over generated answer tokens, for question $q$ and passage $d$. Let $\varnothing$ denote the empty passage, and let $L$ be a set of late layers.
$$ s(d) = \frac{1}{|L|}\sum_{l\in L}\Big(\log\|h_l(q,d)\| - \log\|h_l(q,\varnothing)\|\Big) $$
Subtracting the passage-free run cancels properties of the question and its tokens that inflate norms regardless of the evidence. The reader answers from the passage with the highest $s(d)$. It abstains if every $s(d)$ falls below a threshold $\tau$. The sign of the score, the layer set, and $\tau$ are fit on a small development split and then frozen.
The contribution is twofold. The first part is the measurement: how does the norm shift behave across gold, same-entity-wrong-section, sibling-entity, and random passages? None of the four source directions has examined this. The second part is a scoring rule that competes directly with question-regeneration scoring on the passages where that scoring should fail.
Research Program
The first stage is a controlled noise suite built on Encyclopedic VQA. The benchmark links each question to a specific section of a fixed Wikipedia snapshot and draws its entities from iNaturalist species and Google Landmarks. Those links let us build four passage types per question:
- the gold section;
- another section of the correct entity's article;
- the matching section of a sibling entity, meaning the same genus or a geographically nearby landmark;
- a random passage.
The sibling passages reproduce the "incorrect entity or context selection" errors that EntRAG identifies as its main remaining failure. All four types are length-matched. Each passage is read alone, so position effects such as evidence being overlooked in the middle of long prompts (Grounding Language Models with External Knowledge Retrieval) cannot confound the result.
On this suite we record full layer-wise norm profiles for open-weight readers, pairing text-only and multimodal members of the same family, for example Gemma3 alongside Qwen3 and Phi-4, which The Tell-Tale Norm already studies. We then ask two questions: which layers separate the four passage types, and does the sign of the shift agree across models?
The second stage runs the frozen scoring rule on realistic retrieval. We use top-K passages from an entity-focused retriever on Encyclopedic VQA and on INFOSEEK. As a bridge to prior text results, we also run top-K DPR passages on Natural Questions, TriviaQA and SQuAD, the settings used by Cho et al..
The third stage is a transfer test. We apply the rule, with no refitting, to SciAux, which pairs ScienceQA items with helpful, irrelevant, or misleading auxiliary context (Thinking in a Crowd).
One secondary question comes almost for free. We already compute the passage-free norm $\|h_l(q,\varnothing)\|$. We check whether it predicts whether the model already knows the answer, which is the retrieval decision that SKR makes with an external nearest-neighbour index of training questions.
Baselines
The naive reference point is a reader that concatenates the top-K passages and answers directly, with no passage scoring and no abstention. That is the setting in which Cho et al. document collapsing accuracy. The oracle ceilings are:
- Encyclopedic VQA with perfect retrieval: 87.0% accuracy with PaLM (Mensa et al.), against 48.8% with Google Lens retrieval.
- INFOSEEK-Human: 45.6% with an oracle entity plus FiD, against 18.2% for CLIP plus FiD (Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?).
These figures come from different generators than ours. We use them only to locate the headroom, and we re-measure both the naive and the oracle condition with our own readers.
The strongest direct competitor is Distraction-aware Answer Selection (DAS) from Cho et al.. DAS reads each passage separately, prompts the reader to return "unanswerable" when it cannot answer, and reweights each candidate answer by the likelihood of regenerating the question from the passage. It is the strongest competitor because it solves the same problem, per-passage answer selection under distraction, with the same per-passage cost profile and without training. The reported figures are an average 64% relative increase in Exact Match over naive zero-shot readers, and a 231% relative gain with 100 DPR documents on SQuAD.
We reimplement DAS on the same readers, the same retrieved passages, and the same development split for its single interpolation weight. The scoring rule counts as beating DAS if it achieves higher answer accuracy on Encyclopedic VQA and INFOSEEK at equal K, and higher sibling-versus-gold detection AUROC on the noise suite, where AUROC is the area under the receiver-operating curve. Matching DAS on text benchmarks while clearly beating it on sibling passages would still support the core claim.
The strongest retrieval-side system is EntRAG. It binds entity tokens to visual features and reranks passages with an entity-aware ranker, reporting 46.1 on Encyclopedic VQA and 44.5 on InfoSeek. It acts before reading, so it complements our reader-side score rather than competing with it. The file does not indicate that its code is available. We therefore train a cross-encoder reranker on Encyclopedic VQA section labels as a stand-in, and ask whether the norm shift adds accuracy on top of that reranker's top-K.
Three further baselines use the same per-passage runs:
- the reader's answer probability;
- the answer-token entropy;
- a logistic probe trained on full late-layer hidden states to predict gold versus distractor, fit on the same development split.
For retrieval decisions we compare against SKR-kNN. It reports a 2.91% gain over full retrieval and 4.08% over chain-of-thought with InstructGPT (Wang et al.). We rebuild it on our readers.
Three ablations isolate the key ingredient:
- the raw norm without subtracting the passage-free run;
- the norm taken at early layers;
- the norm under a token-shuffled copy of the passage, which keeps the words but destroys meaning.
If shuffled passages produce the same shift as intact ones, the signal is lexical rather than evidential.
Why not just…?
Why not just use the reader's answer probability? Cho et al. show that distracted readers assign probability as high as 0.98 to wrong answers, so probability ranks confident mistakes above correct answers. We still run probability as a baseline on the sibling-passage split, since that is where its failure should be sharpest. If probability matches the norm shift there, the norm adds nothing.
Why not just use DAS's question-regeneration score? This is the real threat. The argument against it is specific. A passage about a sibling species describes the same kind of organism, uses the same vocabulary, and regenerates the question nearly as well as the gold passage. Regeneration therefore measures topic, not answerhood. The noise suite tests exactly this. If regeneration scoring separates sibling from gold passages as well as the norm shift does, we will report that the geometric signal is redundant. The remaining question would then only be whether combining the two scores helps.
Why not just train a probe on the hidden states? A supervised probe sees strictly more information than a single norm, so it will likely win in-domain. The case for the norm is transfer. It has one sign, a few layers, and one threshold, and should hold up under shifts between species and landmarks, from vision to text, or to SciAux misleading context, where a probe may learn dataset-specific directions. We fit the probe on Encyclopedic VQA and evaluate both methods on INFOSEEK, the text QA sets, and SciAux. If the probe transfers as well as the norm does, the recommendation becomes "use the probe," and the norm result stands as a mechanistic finding rather than a method.
Why not just improve the retriever or reranker? Better retrieval raises the share of gold passages but does not remove sibling passages. Entity-level ambiguity is intrinsic to fine-grained entities, and even EntRAG attributes most of its remaining errors to them. The stacking experiment answers this directly: if the norm shift adds no accuracy on top of a trained cross-encoder's top-K, reranking is sufficient.
Proposed Experiments
The first decisive check runs on the controlled Encyclopedic VQA suite. We test whether the late-layer shift $s(d)$ separates gold from sibling passages with a sign that is consistent across at least three reader models. We also test whether it survives the length-matching and shuffled-passage controls. If the sign flips between models, or shuffled passages match intact ones, the proposal stops here with a clear negative result.
The second check is end-to-end accuracy on Encyclopedic VQA and INFOSEEK, reported against the naive reader, DAS, probability, the probe, and the reranker stack at K of 5, 20 and 50. We use each benchmark's official accuracy metric. We also report risk-coverage curves for abstention, which show accuracy as a function of the fraction of questions answered.
The third check is transfer. We apply the frozen rule to Natural Questions, TriviaQA and SQuAD, measured by Exact Match against DAS, and to SciAux misleading-context items, measured by accuracy against the naive reader. We compare the frozen rule with the probe under the same shift.
The fourth check is secondary. We measure whether the passage-free norm predicts known versus unknown questions on StrategyQA and TruthfulQA, and whether retrieving only when the norm says the question is unknown matches the accuracy of SKR-kNN.
Compute is matched throughout. Every per-passage method, including DAS, uses K+1 forward passes per question, and every tuned method gets the same development split.
References
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- MagFace: A Universal Representation for Face Recognition and Quality Assessment
- AdaFace: Quality Adaptive Margin for Face Recognition
- The Tell-Tale Norm: $\ell_2$ Magnitude as a Signal for Reasoning Dynamics in Large Language Models
- Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories
- EntRAG: Entity-Centric Retrieval-Augmented Generation for Knowledge-based Visual Question Answering
- U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-in-a-Haystack
- Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?
- Thinking in a Crowd: How Auxiliary Information Shapes LLM Reasoning
- Self-Knowledge Guided Retrieval Augmentation for Large Language Models