Most retrieval-augmented language models fetch the same amount of text for every question. They run a search, paste the top few passages into the prompt, and generate. Recent "adaptive retrieval" methods ask a yes-or-no question first: does the model need to look anything up at all? This proposal argues that the yes-or-no question is too coarse. A model that half-knows a fact may need one sentence to fix its answer, while a model that knows nothing about an obscure entity may need a full passage. We propose to measure, question by question, the smallest amount of retrieved text that turns a wrong answer into a right one. We then use that measurement to test a specific prediction and to build a gate that chooses how much to retrieve, not only whether to retrieve.
The prediction comes from coding theory for distributed storage. There, a failed disk is rebuilt by downloading data from surviving disks. Classic results show two things. First, how much must be downloaded depends on how much each surviving disk already stores. Second, insisting that the rebuilt disk hold an exact copy of the lost data costs more than insisting only that the system stay decodable, and that extra cost shows up in the middle of the tradeoff rather than at its ends. Our hypothesis is that retrieval behaves the same way. Feeding the model whole, fixed chunks from a static index (the "exact" analogue) wastes tokens compared with feeding it the minimal set of facts that suffices (the "functional" analogue). That waste should concentrate on questions the model partly knows, not on questions it knows well or not at all.
Where the analogy comes from, and how far we push it
Network Coding for Distributed Storage Systems set up the problem of rebuilding a failed storage node by downloading $\beta$ units from each of $d$ helper nodes, each of which stores $\alpha$ units. It showed a tradeoff curve between per-node storage and repair download. It also showed that this curve is reachable under "functional repair," where the replacement node only has to keep the file recoverable, not reproduce the lost bits. Two later results constrain what we should expect when exactness is required. Characterizing the Rate Region of the (4, 3, 3) Exact-Repair Regenerating Codes used a computer-aided linear-programming proof to show that exact repair has a strictly worse boundary than functional repair: $4\bar\alpha + 6\bar\beta \ge 3$ against the functional $8/3$. Distributed Storage Codes With Repair-by-Transfer and Nonachievability of Interior Points showed that exact repair can reach the two endpoints of the curve but not many of the points between them. It traced the failure to helpers that must serve many possible failure patterns with one fixed layout.
We use these papers as a design lens that generates falsifiable predictions, not as a source of bounds. Tian's proof technique only scales to tiny systems, and language-model knowledge is not a linear code. So we make no information-theoretic claim about LLMs. The mapping is deliberately loose. The model's parametric knowledge about a question plays the role of what the "surviving nodes" already hold. Retrieved tokens play the role of repair download. A fixed chunk index plays the role of exact repair: a static layout built in advance for every possible query, much like a helper's stored contents must serve every possible failure. The question we ask is empirical: does the interior-point penalty reappear?
What adaptive retrieval already does and leaves open
The retrieval side motivates the work. When Not to Trust Language Models built PopQA and showed that parametric recall tracks entity popularity. It also showed that retrieval can hurt on popular facts: in about 10% of cases, retrieval made GPT-3 davinci-003 answer incorrectly. Gating retrieval by a popularity threshold improved accuracy and cut cost. TARG decodes a 20-token draft and gates retrieval on the gap between the top two token probabilities. Because instruction-tuned models have sharply peaked output distributions, this gap stays informative where entropy does not. TARG reports cutting retrieval frequency by 70–90% without losing accuracy. White-box gates such as SeaKR, CtrlA and Probing-RAG read hidden states instead of output probabilities. Self-RAG trains the model to emit retrieval tokens. All of these make the decision binary, or a sequence of binary decisions. Once they fire, they retrieve a fixed top-$k$ of fixed chunks.
The granularity side supplies the method. PropRAG splits documents offline into atomic, self-contained propositions and searches over them. KAR³-RAG splits both the corpus and the query into atomic units. These methods shrink the unit of retrieval, but they apply the same granularity to every query. They also never ask how much evidence a given model actually needed. KAR³-RAG also reports a constraint we must respect: atomic pipelines collapse with weaker reasoners. So we keep the model in the loop simple and let a cheap search choose the evidence.
Core Claim
The amount of retrieved text a model needs is a per-query quantity that depends on how much the model already knows. Fixed-chunk retrieval overpays relative to minimal proposition-level evidence, and the overpayment is largest for partially known facts. If both hold, a gate that predicts a retrieval budget should match the accuracy of the best binary gate while sending substantially fewer context tokens. A budget here means none, one or a few propositions, or full passages. Cheap uncertainty signals such as TARG's margin should be enough to predict it.
Research Program
The first component is an oracle measurement of "repair bandwidth." For each PopQA question that a fixed model gets wrong without retrieval, we take the gold Wikipedia passage and the top retrieved passages and split them into propositions in PropRAG's style. We then search for the smallest set of propositions that makes the model answer correctly. The search adds propositions greedily by retriever score, with a small exhaustive search over pairs to check whether greedy ordering misses smaller sets. We record the cost in tokens. We separately record the smallest number of whole chunks that achieves the same flip. Their ratio is the "exactness overhead" for that question.
We plot both quantities against entity popularity and against the model's no-retrieval margin. Comparing Llama-3.1-8B with larger models from the same family gives a second axis for parametric capacity. This yields an empirical storage-versus-download curve for retrieval, at much finer resolution than the accuracy-versus-scale plots in Mallen et al.
The second component is a budgeted gate. TARG's draft-and-margin signal is cheap and works without access to hidden states, so we use it as the uncertainty input. A small calibration model maps it to a budget level: none, propositions up to a token cap, or full top-$k$ chunks. The calibration model is trained on the oracle labels from the first component, using a held-out split. Because it consumes only output probabilities from a short draft, it does not inherit the white-box dependency of SeaKR or CtrlA.
The third component tests transfer. We train the calibration on PopQA and evaluate it unchanged on TriviaQA and NQ-Open. On HotpotQA and 2WikiMultiHopQA, we check whether the budget idea survives when evidence must be combined across documents. In that setting the functional advantage may shrink, because each proposition loses the context that links it to the others.
Baselines
The naive reference points are Never-RAG and Always-RAG. Always-RAG means fixed top-$k$ dense retrieval of whole chunks on every question. Both run on the same model, corpus, retriever and decoding settings as every other system.
The strongest binary competitor on PopQA is popularity-threshold adaptive retrieval from When Not to Trust Language Models. It reports up to 46.5% accuracy with GPT-3 davinci-003, a 5.3% improvement over non-adaptive methods. We re-run it on our model with thresholds tuned per relation type on the same calibration split we use.
The strongest training-free gate is TARG with the margin signal. It reports higher exact match than Always-RAG on PopQA with 70–90% fewer retrievals. No raw EM value for TARG on PopQA was found on Lacuna, so we reproduce it under our setup and tune its threshold with the same budget we give our calibration model.
As a granularity baseline we run PropRAG-style proposition retrieval on every question, with no gate. As a white-box reference on TriviaQA, where SeaKR reports 63.1 F1, we include SeaKR's numbers for context but do not treat them as directly comparable.
Two ablations isolate the key ingredient, which is sizing. The first is a matched-budget binary gate: TARG's gate firing proposition retrieval at a fixed size chosen so that average tokens equal ours. If this matches us, the gain came from propositions, not from per-query sizing. The second is a shuffled-budget control that keeps our budget distribution but assigns budgets at random. If this matches us, the uncertainty signal carries no sizing information.
Beating the baselines means two things at once. First, we must match TARG's exact match within confidence intervals while using at least 30% fewer context tokens per retrieved query. Second, we must beat the matched-budget binary gate, on several random seeds of the calibration split.
Why not just…?
Why not just use a binary gate like TARG, since it already removes most retrievals? This is the main threat. If most of the cost of RAG is in issuing searches rather than in the tokens prefilled once a search fires, sizing adds little. We will report both search calls and prefilled tokens, along with time-to-first-token. If the oracle shows that queries passing the gate typically need nearly full passages, we will report sizing as unnecessary. That finding is still worth having, because it would say binary gating captures essentially all of the available savings.
Why not just always retrieve propositions, as PropRAG does? PropRAG reports 500–1000 ms retrieval latency, against roughly 50 ms for plain dense retrieval. Our hypothesis also predicts that propositions alone will underperform on long-tail entities, where the model needs surrounding context. The always-propositions baseline, broken down by popularity bin, tests this directly. If propositions win in every bin, the interior-concentration prediction is false and we will say so.
Why not just train a classifier to predict top-$k$, or compress retrieved context after the fact? A learned predictor of $k$ is essentially our calibration model with chunks as the unit. We include it as a variant, so this alternative is part of the experiment rather than something we argue against. Post-hoc compression still pays the full search and reading cost, and none of the cited papers report a compressor matched against oracle-minimal evidence. The distinctive claim of the proposal is the oracle measurement and the predicted shape of the overhead, not the predictor architecture.
Decisive Checks
The idea stands or falls on three results. First, the oracle repair bandwidth must vary strongly within the set of questions where retrieval helps. If nearly all of them need the same amount, there is nothing to size. Second, the exactness overhead must peak in middle popularity bins and shrink at both ends, echoing the interior-point result of Shah et al. If the overhead is flat or largest at the tail, the storage analogy is not predictive, and we will drop it in favor of the purely empirical claim. Third, the budgeted gate must beat the matched-budget binary ablation on PopQA and keep that advantage when transferred unchanged to NQ-Open and TriviaQA. Failure on multi-hop sets is an acceptable, reportable limitation. Failure on single-hop transfer would mean the sizing signal is a PopQA artifact.
References
- Network Coding for Distributed Storage Systems
- Characterizing the Rate Region of the (4, 3, 3) Exact-Repair Regenerating Codes
- Distributed Storage Codes With Repair-by-Transfer and Nonachievability of Interior Points on the Storage-Bandwidth Tradeoff
- When Not to Trust Language Models: Investigating Effectiveness and Limitations of Parametric and Non-Parametric Memories
- TARG: Training-Free Adaptive Retrieval Gating for Efficient RAG
- SeaKR: Self-aware Knowledge Retrieval for Adaptive Retrieval Augmented Generation
- CtrlA: Adaptive Retrieval-Augmented Generation via Inherent Control
- Probing-RAG: Self-Probing to Guide Language Models in Selective Document Retrieval
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
- PropRAG: Guiding Retrieval with Beam Search over Proposition Paths
- From Complex to Atomic: Enhancing Augmented Generation via Knowledge-Aware Dual Rewriting and Reasoning