Retrieval-augmented language models look up passages and paste them into the prompt before answering. They have an awkward failure: adding more passages often makes answers worse. Irrelevant or misleading text pulls the model's attention away from the evidence that matters. This note proposes a different rule for deciding what goes into the prompt. A passage earns its place only if it changes what the reader model believes the answer is, per token of prompt space it consumes. Passages are then added one at a time, and after each addition every remaining passage is re-scored against the reader's updated belief. Selection stops when no remaining passage beats a statistically calibrated noise floor.
The rule comes from subjective interestingness, a line of data-mining work that scores a pattern by how surprising it is under a model of what the analyst already believes, divided by how costly it is to describe. The analyst's belief model is then updated after each reported pattern, so redundant patterns lose their score (Maximum entropy models and subjective interestingness: an application to tiles in binary databases). We move that machinery from a human analyst mining a database to a language model reading retrieved text. The "analyst" becomes the reader model's distribution over candidate answers. The "description cost" becomes the number of prompt tokens. The payoff is a single training-free, generator-agnostic procedure with three properties. It retrieves nothing when the model already knows the answer. It skips passages that repeat what earlier passages already said. It stops before the long-context accuracy collapse sets in.
Core Claim
Two things usually decide which passages go into the prompt: how similar a passage looks to the question, or a learned judgment of what "helps." Neither accounts for what the reader already believes. We claim that belief shift per token is the better quantity to select on. Belief shift here means how far a passage moves the reader's distribution over answers, given the passages already chosen. We also claim it separates the two kinds of noise documented in the literature. Hard negatives are passages that resemble the query but lack the answer, and they drive the accuracy decline seen with strong dense retrievers (Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG). We predict they will score high on embedding similarity but low on belief shift, because they do not supply a determinate answer. Unnecessary retrieval for questions the model already knows is the case Self-Knowledge Guided Retrieval Augmentation for Large Language Models handles with a separate classifier. We predict it will show up as near-zero belief shift for every candidate, so the procedure stops at zero passages without any separate gate.
Concretely, let $P_S(a)$ be the reader's probability of answer $a$ given the question and the selected passage set $S$, and let $|d|$ be the token length of candidate passage $d$. The score is
$$SI(d \mid S) = \frac{\mathrm{KL}\left(P_{S \cup \{d\}} \,\|\, P_S\right)}{|d|},$$
which is information gained about the answer per prompt token. It mirrors the information content over description length ratio of Maximum Entropy Models for Iteratively Identifying Subjectively Interesting Structure in Real-Valued Data. The difference is that surprise is measured on the answer belief, not on the passage text. This distinction matters. If surprise were measured as the passage's own improbability under the language model, off-topic text would score highest.
Relation to Prior Work
The subjective interestingness papers supply the method. They contribute the surprise-over-cost ratio, the update-then-rescore loop that makes redundant patterns lose their score, and the idea that falling scores give a principled place to stop (Maximum entropy models and subjective interestingness: an application to tiles in binary databases; Maximum Entropy Models for Iteratively Identifying Subjectively Interesting Structure in Real-Valued Data). In those papers the belief model is an explicit maximum-entropy distribution over a data table. Ours is implicit: the reader model itself, queried for answer probabilities. We give up the closed-form model and gain a belief state that is exactly the one that determines the answer.
Three lines of retrieval work motivate the gap. QALinkPlus: Text Enrichment with QA data re-ranks retrieved QA pairs for "interestingness" using hand-written graph rules for novelty, complexity, and hub entities. It never measures a reader's beliefs or its downstream distraction, and we replace its rules with a probabilistic quantity tied to the reader. CompAct: Compressing Retrieved Documents Actively for Question Answering reads documents in segments and stops when a fine-tuned model emits a completion token. We aim for the same adaptive stopping without supervision and with a stopping rule that has a statistical meaning. SKR gates retrieval with a binary known/unknown classifier built by nearest-neighbor lookup over past questions, because direct self-assessment prompting was the least effective method it tested (Self-Knowledge Guided Retrieval Augmentation for Large Language Models). That negative result constrains our design. We never ask the model whether it knows. We measure whether its answer distribution moves.
Familiarity-aware Evidence Compression for Retrieval Augmented Generation appears to contradict us. It finds that readers use compressed evidence best when that evidence is familiar, meaning low perplexity under the reader. Under subjective interestingness, fully familiar content carries no information. We resolve the conflict by measuring the two quantities on different objects. Familiarity is a property of the surface wording, while information is a property of the answer belief. The secondary analysis below tests whether both matter independently.
Diagnostic work constrains what "success" means. Lost in the Middle: How Language Models Use Long Contexts and Long-Context LLMs Meet RAG show position effects and an accuracy curve that rises then falls with passage count. A selection rule should therefore be judged on accuracy as the candidate pool grows, not at one fixed number of retrieved passages.
Research Program
Estimating the answer belief. For each question, we build a candidate answer set from several sources: the reader's closed-book samples, its greedy answer under each single candidate passage, and its greedy answer under the current selected set. Answers are normalized by string matching. We then score each candidate by teacher-forced log-likelihood under the reader and renormalize over the set to obtain $P_S$. This needs only token log-probabilities, so it works for any reader and any tokenizer. That is a direct contrast to FAVICOMP's shared-tokenizer requirement. For readers that do not expose log-probabilities, we fall back to sample frequencies over $K$ generations and report how much this approximation costs.
Selection loop. The retriever returns a pool of $M$ candidates, with $M$ from 10 to 50. At each step we score every remaining candidate conditioned on $S$ and add the highest scorer. We stop when the best score falls below a floor $\tau$. Following the randomization logic behind the interestingness literature, $\tau$ is set from a null distribution. We compute $SI$ for passages retrieved for other, unrelated questions and take the 95th percentile. The floor is therefore "indistinguishable from irrelevant text" rather than a tuned constant.
Multi-hop lookahead. Greedy selection has a known weakness for multi-hop questions. A bridge passage alone may not move the answer at all, and only becomes informative together with a second passage. This is exactly how single-step compressors lose bridge entities (CompAct). We therefore add a bounded lookahead. At each step, the top few candidates are also scored as pairs, and a pair is admitted if its joint score clears the floor. We report the cost this adds.
Secondary analysis: familiarity versus information. Across all (question, passage) pairs, we regress reader correctness on two variables: belief shift and surface perplexity of the passage under the reader. This tells us whether FAVICOMP's familiarity effect survives once information content is controlled for. If it does, we add a mild perplexity term to the denominator.
Baselines
The naive reference point is standard top-$k$ retrieval. The reader receives the top $k$ passages from the same retriever, for $k$ in {0, 5, 10, 20, 50}. Closed-book ($k=0$) is included because Lost in the Middle reports that GPT-3.5-Turbo on 20-document QA falls from about 75% accuracy with the answer first to roughly 53% with it in the middle, below closed-book. Beating the best single $k$ chosen in hindsight counts as a minimal success. Beating every fixed $k$ at matched or lower average token count is the real target.
The strongest cheap competitor is a cross-encoder relevance reranker with top-$k$ or a score threshold. The file gives no citation or reported number for one, so none is claimed here. It runs over the identical candidate pool, with its threshold tuned on the same development split used to check our null-derived floor. This comparison isolates whether belief shift carries information beyond query-passage relevance.
The strongest training-free existing method is FAVICOMP. It reports a 28.1% accuracy improvement over the best training-free alternative on MuSiQue with Llama3 (Familiarity-aware Evidence Compression). We run it with the same Llama3 reader and the same retrieved pool. Beating it means higher accuracy on MuSiQue at equal or lower prompt tokens and latency, without its shared-tokenizer constraint.
The strongest trained method is CompAct, which reports a 7.0 F1 improvement over RECOMP on HotpotQA (CompAct). The file does not give its absolute HotpotQA score (no reported result found on Lacuna for CompAct's absolute HotpotQA F1), so we reproduce it with its released compressor on our pool. We also run SKR's nearest-neighbor retrieval gate followed by the reranker. That combination is the natural "just combine existing parts" competitor to our zero-passage stopping. It reports a 4.20% improvement over full retrieval with gpt-3.5-turbo (SKR).
Ablations isolate the ingredients. The first replaces KL belief shift with the reader's confidence in its top answer, which tests "information gain" against "certainty." The second removes the token-length denominator. The third scores every passage once, independently, without the update-then-rescore loop, which tests whether redundancy suppression matters. The fourth replaces the null-derived floor with a tuned constant. Every system gets the same reader, the same pool, and the same development-set tuning budget, and we report tokens and wall-clock time alongside accuracy.
Why not just…?
Why not just use a strong reranker and keep the top few? This is the most serious threat. On single-hop NQ it may match us. The evidence against it as a complete answer is that similarity is exactly what hard negatives exploit: stronger dense retrievers degrade the reader faster than BM25 (Long-Context LLMs Meet RAG). A reranker also cannot tell that a relevant passage is useless because the reader already knows its content. The pool-scaling experiment settles the question. If the reranker's curve stays as flat as ours as the pool grows from 10 to 50 e5-retrieved passages, and matches us on the known-question subset, then belief shift adds nothing and we will say so.
Why not just pick passages that make the reader most confident? Confidence is cheaper to compute and close in spirit. However, a confidently wrong reader stays confident, and a passage that confirms an already-known answer raises confidence while adding nothing. The confidence ablation measures exactly this difference, especially on the SKR-style known/unknown split and on distractors that plant a plausible wrong answer.
Why not just gate with SKR and stop there? SKR decides only whether to retrieve at all. Once retrieval triggers, it sends all raw passages to the reader (SKR). The SKR-plus-reranker baseline tests whether a binary gate combined with relevance recovers our gains. We expect a gap on multi-hop sets, where the useful number of passages varies from question to question.
Why not just train a compressor or stopping model, as CompAct and Ext2Gen do? Trained methods can be strong. Ext2Gen reports an average score of 0.557 against 0.473 for the default Llama3.1-8b-instruct (Aligning Extraction and Generation for Robust Retrieval-Augmented Generation). Their cost is that they are tied to the model they were trained for, as PRCA's per-generator retraining illustrates (PRCA). Our claim is not to beat them outright. The claim is to come within a few points of them with no training, and to transfer unchanged to a reader with a different tokenizer. The transfer experiment below tests that.
Proposed Experiments
Benchmarks and readers. We use NQ and TriviaQA for single-hop questions and for splitting questions into ones the model already knows and ones it does not. For multi-hop we use the HotpotQA distractor setting (2 gold plus 8 distractor paragraphs, where BiDAF reached 58.28 answer F1 against 91.40 for humans, per HotpotQA), the HotpotQA full-wiki setting, MuSiQue, and 2WikiMultiHopQA. We also use the 20-document positional setup from Lost in the Middle. Readers are Llama3-8B and Mistral-7B, plus one model from a different tokenizer family for transfer. Metrics are exact match and F1, average prompt tokens, and latency.
Decisive check 1: pool scaling. Plot accuracy against pool size (10 to 50) for e5 and BM25 pools. The prediction is that fixed-$k$ and reranker curves follow the documented rise-then-fall, while ours stays flat. This is because hard negatives fall below the floor. We will directly measure the fraction of annotated non-gold HotpotQA distractors that get admitted. If hard negatives turn out to score high on belief shift, the core claim fails for that noise type, and we will report which distractors fool the method. We expect the problem cases to be passages that name a plausible wrong answer.
Decisive check 2: knowledge-aware stopping. On questions where the closed-book reader is already correct, the method should admit zero or one passage far more often than it does on questions where the reader is wrong. It should match or beat SKR's reported gain over full retrieval without any classifier.
Decisive check 3: multi-hop and lookahead. On MuSiQue and HotpotQA, compare greedy selection against pairwise lookahead, both against CompAct reproduced on the same pool and against FAVICOMP. The gap between greedy and lookahead measures how often bridge passages are invisible to single-passage belief shift.
Decisive check 4: transfer. Take the floor and every setting chosen for Llama3 and apply them unchanged to the different-tokenizer reader. Compare the accuracy drop to that of CompAct's compressor used off-model.
The familiarity regression is reported alongside these checks. It is informative whichever way it comes out, because it tests FAVICOMP's hypothesis once information content is held fixed.
References
- Maximum entropy models and subjective interestingness: an application to tiles in binary databases
- Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG
- Self-Knowledge Guided Retrieval Augmentation for Large Language Models
- Maximum Entropy Models for Iteratively Identifying Subjectively Interesting Structure in Real-Valued Data
- QALinkPlus: Text Enrichment with QA data
- CompAct: Compressing Retrieved Documents Actively for Question Answering
- Familiarity-Aware Evidence Compression for Retrieval-Augmented Generation
- Lost in the Middle: How Language Models Use Long Contexts
- Aligning Extraction and Generation for Robust Retrieval-Augmented Generation
- PRCA: Fitting Black-Box Large Language Models for Retrieval Question Answering via Pluggable Reward-Driven Contextual Adapter
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering