Searching for the Right Context, Starting from a Cheap Guess

Retrieval-augmented language models read a handful of retrieved passages before answering, and how many and which passages they read matters a great deal. Recent work shows that most retrieved passages contribute nothing, that some actively hurt, and that step-by-step "thinking" models are especially fragile. Given two misleading snippets, they reason their way into wrong answers instead of discarding the bad evidence. Current selectors score each passage once with a cheap model and keep those above a threshold. That approach cannot see interactions between passages: two facts that are useless alone and decisive together, two passages that say the same thing, or one passage that quietly contradicts the rest. The only judge that actually sees these interactions is the reader model itself, and asking the reader to evaluate every candidate subset is far too expensive.

This note proposes treating context selection as a small search problem. The reader model is the expensive objective, and a cheap passage scorer supplies the starting point. The idea comes from warm-start optimization, where a fast heuristic hands an exact solver a good first solution so the solver spends its budget improving it rather than finding it. Here a lightweight scorer proposes an initial passage set and also orders the candidate edits. A short local search then adds, drops, or swaps passages using a few cheap "probe" calls to the reader. The final context size is whatever the search settles on, so the window grows for hard multi-hop questions and shrinks for easy ones. The probe score includes a label-free signal of trouble: misleading context makes reasoning models produce abnormally long, conflicted outputs. The search can therefore detect and drop passages the reader cannot reconcile, before the full reasoning pass ever sees them.

Core Claim

A fixed, small budget of reader probes, perhaps four to eight short generations per query, spent on a local search that starts from a cheap scorer's selection, will choose better contexts than either the cheap scorer alone or the same probe budget spent without a good starting point. The advantage should concentrate on two kinds of questions: multi-hop questions where evidence is complementary, and questions with topically relevant but misleading snippets. On those, single-passage scores are wrong in a structured way.

The claim has two separable parts, and both can be falsified. The first is that the reader-in-the-loop objective captures passage interactions that independent scores miss. The second is that warm-starting is what makes a tiny probe budget sufficient. If cold-start search with the same budget does as well, the warm-start part of the claim fails. If the warm start alone does as well, the search adds nothing.

Where This Sits

Three lines of work motivate the gap. Influence Guided Context Selection for Effective Retrieval-Augmented Generation measures each passage's leave-one-out effect on the generator and trains a small surrogate to predict it. It found that roughly 80% of retrieved passages have near-zero influence. It still keeps passages independently whenever predicted influence is positive. EXIT classifies sentences in document context and keeps those above a threshold, and also scores them independently. QUITO scores tokens by the attention a 0.5B model pays from the query and compresses to a fixed ratio. We reuse these three as the warm-start heuristics rather than compete with them. Our question is what a small amount of reader feedback adds on top of them.

Two works already move beyond independent scoring, and they are the closest prior art. Shifting from Ranking to Set Selection for Retrieval Augmented Generation (SETR) has a fine-tuned 8B model enumerate a query's information requirements and pick a covering subset in one generative pass. FrugalRAG trains a policy with reinforcement learning to stop iterative retrieval at the right number of hops. Neither one queries the downstream reader about the subset it actually chose. SETR's selector reasons about coverage without ever observing whether the reader is confused by the selection. FrugalRAG decides when to stop gathering, not which already-gathered passages to discard. Our method is training-free on the selection side and uses the reader's own behavior as the objective.

Two results constrain the design. Thinking in a Crowd shows on SciAux that two misleading snippets drop Qwen3-8B with thinking enabled from 96.22% to 77.03%, below thinking-off at 74.87%, and that these cases produce the longest outputs. This is why the probe objective includes output length as a conflict signal. It is also a warning: a naive "reader confidence" objective could reward confidently wrong contexts. SETR's argument that chain-of-thought is needed to judge coverage seems to pull the other way, since it puts deliberate reasoning before selection. We resolve this by keeping probes short and budget-limited and reserving full reasoning for the final pass, so probes never run long enough to commit to a rationalization.

The warm-start lens comes from optimization work that seeds expensive solvers with cheap solutions. FMIP generates mixed-integer solutions that Gurobi then improves, reducing primal gaps by 41.34% on average at under 1% overhead. Start Smart initializes explanation-mask optimization from quantile-transformed gradients and converges up to 6.73x faster than an all-ones start. We borrow the principle and the evaluation habit: always compare against a cold start at equal compute. We do not borrow either method directly.

We deliberately do not cast selection as an integer linear program solved exactly. With a pool of about twenty passages and selections of two to six, exact optimization over surrogate scores is trivial. The hard part is that the true objective is expensive and non-additive.

Research Program

For each query, a standard retriever returns a pool of twenty candidate passages. A cheap scorer assigns each passage a utility. We test three scorers: an influence surrogate in the style of the CSM, an EXIT-style classifier, and QUITO-style query attention. The warm start $S_0$ is the scorer's thresholded set.

A probe evaluates a candidate set $S$. The reader runs with thinking disabled or a small token cap, producing $m$ sampled short answers. We record the fraction $a(S)$ agreeing with the majority answer and the mean output length $\ell(S)$, normalized by the query's length on $S_0$. The probe score is

$$J(S) = a(S) - \alpha\,\ell(S) - \beta\,\frac{|S|_{\text{tok}}}{T},$$

where $|S|_{\text{tok}}$ is the token count of $S$, $T$ is the pool's total tokens, and $\alpha, \beta$ are two scalars tuned once on a development split and then frozen.

Search is first-improvement local search over add, drop, and swap moves. Moves are ordered by the cheap scorer: drop the lowest-scored member, add the highest-scored non-member, and swap the weakest member for the strongest outsider. A move is accepted when $J$ improves. The search stops at a budget of $B$ probes or when no move improves. The selected set then goes to the reader for one full reasoning pass. We sweep $B \in \{0, 2, 4, 8, 16\}$, where $B=0$ is the warm start alone.

The analysis component is as important as the method. We will measure where the search changes the warm start and whether those edits correlate with three things: passages SciAux labels as misleading, gold supporting facts on HotpotQA and MuSiQue, and passages near-duplicate to another member. These checks show whether the search is detecting interactions or just adding noise. We will also test the length signal directly. On SciAux, does $\ell(S)$ separate sets containing misleading snippets from clean sets, with and without thinking enabled in the probe?

Baselines

The naive reference is fixed top-5 passages passed to the reader with full reasoning. This is the configuration every dynamic-windowing paper argues against. SETR reports that RankZephyr's top-5 uses 2,672 tokens on MultiHopRAG. We include a second naive reference that passes the entire twenty-passage pool, which tests the "just use the long window" position.

The strongest existing selector is SETR, because it is the only method in the file that selects a variable-size set with interactions in mind. It reports 47.14% accuracy on MultiHopRAG using about 1,240 generator tokens and about 2.9 passages, versus 45.69% for RankGPT (SETR). We will run SETR's selector on the same retriever pool and the same reader as our method. Beating it means higher MultiHopRAG accuracy at no more total reader-side tokens, with probe tokens counted. Probe tokens are charged to our method in every comparison, and SETR's 8B selector tokens are reported alongside.

For iterative retrieval we compare against FrugalRAG, which reports 68.47% model-based evaluation on HotpotQA with 2.05 searches, against ReAct's 64.20% with 2.76 searches (FrugalRAG). Its setting differs, since it retrieves iteratively rather than selecting from a fixed pool. We will therefore report a variant that runs our search over the union of passages FrugalRAG's explore stage collects, and treat beating it as a secondary target.

The independent-scoring selectors are the CSM-style influence threshold, EXIT, and QUITO, each run with the authors' recommended thresholds or ratios. The file gives an aggregate 15.03% gain for CSM and a 3.7-point EM gain for EXIT. It gives no HotpotQA-specific number for either: no reported result found on Lacuna for CSM or EXIT on HotpotQA alone. QUITO reports 58.9% on NaturalQuestions at 2x compression, and we use NQ as a single-hop control where we expect little gain.

Two ablations isolate the key ingredients, and both use exactly the same probe budget $B$. The cold-start ablation runs the same search from the full top-5, then from a random subset of size $|S_0|$, with moves ordered randomly. This tests whether the heuristic seed matters. The objective ablation removes the length term ($\alpha = 0$) and then the agreement term. A final reference uses gold answers inside $J$ as an oracle objective, giving an upper bound for what the search could reach with a perfect judge.

Why not just…?

Why not just use the cheap scorer's threshold, as CSM and EXIT do? This is the $B=0$ point of our sweep, so the experiment answers the question directly. The prior argument against it is structural. A passage holding only the bridge entity of a two-hop question has low standalone influence but high joint influence. A misleading SciAux snippet is topically relevant and will score high under any relevance model. If the curve from $B=0$ to $B=8$ is flat on HotpotQA, MuSiQue and SciAux, the search is not worth its cost and we will report that.

Why not just apply redundancy-aware greedy selection, in the style of maximal marginal relevance, with no reader calls? Greedy diversity selection handles duplicates cheaply, and on 2WikiMultiHopQA-style redundancy it may capture most of the gain. It cannot detect contradiction or complementarity, because both require knowing what the reader concludes. We include it as a baseline at zero probe cost. If it matches our method outside SciAux, we will narrow the claim to misleading-context robustness.

Why not just spend the probe budget on self-consistency over the full context? This is the most serious alternative. Majority voting over several full-context reasoning samples uses the same extra compute and needs no search. The SciAux evidence suggests it fails when misleading snippets are present, because the 77.03% figure reflects a systematic shift toward wrong answers that voting would reinforce rather than cancel. That is an inference, not a measured result. We will run full-context self-consistency at matched total tokens on every benchmark, and on SciAux it is the comparison that decides the paper.

Why not just train a selector end to end, like SETR or a FrugalRAG-style policy? Trained selectors amortize the cost that our probes pay per query and may win on in-distribution latency. Our method needs no selector training and adapts to whichever reader is deployed. That matters because what confuses Qwen3-8B may not confuse a different reader. We test this by swapping readers without retuning, and we report latency honestly. If SETR matches our accuracy at lower latency on MultiHopRAG, our method's value reduces to reader-specificity and robustness to misleading context.

Proposed Experiments

The main experiments use Qwen3-8B as the reader, so the SciAux numbers are directly comparable, and Llama-3.1-8B as a second reader for the transfer test. Benchmarks are SciAux (resource) for misleading context, HotpotQA, MuSiQue and 2WikiMultiHopQA for complementary evidence, MultiHopRAG for the SETR comparison, and NaturalQuestions as a single-hop control. Metrics are accuracy or exact match, final context tokens, total reader tokens including probes, and wall-clock latency. All methods share the same twenty-passage pool, so upstream retrieval misses, a limitation SETR and FrugalRAG both flag, affect everyone equally.

Three checks would settle the proposal. First, on SciAux with two misleading snippets, does the warm-started search with the length term bring thinking-mode accuracy back toward the 96.22% clean baseline? It must clearly beat the 77.03% unfiltered figure and full-context self-consistency at matched tokens. Second, on MuSiQue and HotpotQA, does accuracy rise with $B$ for warm-started search? It must rise faster than for cold-start search at every budget, and the accepted edits must enrich for gold supporting facts. Third, on NaturalQuestions, does the method avoid spending probes, ideally stopping after the first unimproved move with no accuracy cost? If the first check fails while the second succeeds, the contribution is a better interaction-aware selector rather than a defense against misleading context, and we will frame it that way. If cold start matches warm start everywhere, the warm-start framing should be dropped.