Does the Model Read What It Retrieves? Counterfactual Evidence Tests for Search Agents in Expert Domains

Search-augmented language models are sold on one promise. The model does not know today's stock price, the governing statute, or the right theorem, so it looks them up and answers from what it finds. In expert domains the model often does hold an answer already: last year's revenue figure, a half-remembered legal threshold, a plausible formula. It holds that answer with confidence, and the answer is wrong or out of date. We propose to test, with controlled counterfactual edits to what a search agent retrieves, whether its final answer actually follows the evidence or follows its prior guess. When it follows the guess, the retrieved documents and the step-by-step reasoning become decoration, a justification written after the decision was made ("post-hoc rationalization").

The practical stakes are that expert users see citations and a careful-looking chain of reasoning, and take both as proof that the answer came from the source. Our core bet is testable. We claim that a large share of search-agent errors in finance, law and technical science are not retrieval failures. They are cases where the right evidence was fetched and then overridden. We claim this overriding can be predicted cheaply from how confident the model was before it searched, and we claim it explains the unexplained "overthinking penalty" seen when reasoning models do simple financial lookups.

Core Claim

We make one measurement claim and one intervention claim. The measurement claim is that the evidence-following rate falls sharply as the model's closed-book confidence rises, and that turning on extended reasoning lowers it further on confident items. We define this rate as the fraction of items where swapping the key value in the retrieved documents for a plausible alternative moves the final answer to that alternative. The intervention claim is narrower. Forcing the final answer to be computed only from a short evidence note, in which every number or rule must be quoted from a retrieved document, makes faithfulness both higher and checkable without model internals. We will also test whether this still helps once a simple untrained extract-then-answer pipeline is taken into account.

Three prior findings motivate the claim. Unveiling Confirmation Bias in Chain-of-Thought Reasoning shows that when a model's zero-shot answer distribution has low entropy, its reasoning becomes short and confirmatory and overrides contrary intermediate logic. That work is closed-book commonsense QA; we carry the entropy signal into retrieval. Correctness is not Faithfulness in Retrieval Augmented Generation Attributions shows that up to 57% of Command-R+ citations on NaturalQuestions are post-hoc. The model answers from memory and attaches citations by surface overlap. That study is Wikipedia QA with a fixed corpus and does not vary prior strength. Thinking in a Crowd: How Auxiliary Information Shapes LLM Reasoning shows reasoning models building "logical bridges" to justify misleading snippets. That is the opposite failure, where the model over-follows bad context. Our design measures both directions on the same items. Evidence that disagrees with the prior tests override. Evidence that is subtly wrong tests capture.

Research Program

Record-and-replay retrieval. Live search is not reproducible, so we first run each agent on each question with real tools and cache every search result and page it sees. We then replay the session against the cache. At replay time we can edit the cached documents. Retrieval stays "dynamic" in the sense that the agent picks its own queries and reads what it chooses. Only the content returned is controlled. If the agent issues a query that was never cached, we fall back to live search and mark the item. Items with heavy fallback are analysed separately.

Counterfactual edits by domain. Each edit targets the single fact the answer depends on.

  • Finance. We use FinSearchComp, 635 expert-curated questions across T1 real-time, T2 simple historical and T3 multi-hop tiers. We change the target figure in every cached source to a value 10–30% away, applied consistently across documents so that the swap is internally coherent.
  • Technical science. We use TheoremQA with a theorem-lookup tool. We alter a constant, exponent or condition in the retrieved theorem statement and recompute the gold answer the altered theorem implies. This directly addresses an open question: did theorem augmentation fail there because models ignored the definitions, or because they fit their own calculation into them?
  • Law. We use a subset of AppealCase with a statute-lookup tool. We restrict edits to provisions with a numeric or bright-line condition, such as limitation periods or monetary thresholds, where a legal annotator can confirm the edit flips the correct outcome.
  • Static control. SciAux serves as a static-context control whose clean and misleading conditions are already published.

Confidence without internals. Closed-book confidence is measured as token-level answer entropy for open-weight models. For API models we use agreement among 10 closed-book samples, so the analysis also covers black-box systems. We report the evidence-following rate in confidence bins. We report it separately for edits that move evidence away from the model's closed-book answer (tests override) and edits that move it toward a wrong value the model did not already believe (tests capture).

The overthinking test. FinSearchComp reports that reasoning modes hurt on T1 lookups but offers no mechanism. We run the same open-weight models with thinking on and off on identical replayed sessions. If the penalty is confirmation bias, it should concentrate in high-confidence T1 items where the retrieved value contradicts a stale prior. There the thinking-on trace should contain the retrieved number and then argue it away. If instead the penalty is spread evenly across confidence bins and comes from extra, worse queries, our hypothesis is wrong and we report that.

The intervention. The agent keeps its search loop but must end with an evidence note of at most ~150 tokens, in which every number and every quoted rule is a verbatim span from a cached document. A string check verifies this for free. A separate answer step then sees only the question and the note. The documents and the reasoning trace are removed from its context.

This borrows the bottleneck idea from Markovian Transformers for Informative Language Modeling, which discards the question and answers from the chain of thought alone. That work shows such traces become load-bearing and stay within 3–4 points of unconstrained models, but only on GSM8K, ARC and arithmetic, never with retrieval. It also borrows from FaiRR, whose composer sees only selected rules and facts, a design limited to synthetic rulebases.

We train the note writer and answerer with group-relative policy optimization, following the Markovian recipe. The reward is correctness minus a penalty for ungrounded spans, and a KL penalty discourages hiding information in phrasing. Unlike Markovian Transformers we keep the question in the answer step, and we run an ablation that drops it. Dropping the question is impossible for some legal and finance queries and would confound the comparison.

Baselines

Naive reference. The same model runs the same replayed sessions with standard prompting and no instruction beyond "answer the question using search." This fixes the baseline evidence-following rate and accuracy per tier. For context, FinSearchComp reports that models without search score 0% on T1 and that financial plugins add up to 31.9 points. It names Grok 4 (web) and DouBao (web) as leaders but gives no per-model accuracy; no reported result found on Lacuna for the top system's FinSearchComp accuracy. TheoremQA reports GPT-4 with program-of-thought at 51%.

Instructed grounding. The prompt says "answer only from the retrieved sources and quote the figure you use." This is the cheapest fix a practitioner would try. It gets the same number of samples and prompt-tuning trials as our method: five prompt variants selected on a held-out 10% split.

SYNCHECK with faithfulness-oriented decoding. Synchronous Faithfulness Monitoring for Trustworthy Retrieval-Augmented Generation monitors likelihood, uncertainty, the KL divergence between output distributions with and without context, and semantic alignment. It backtracks on sentences it flags. It is the strongest existing competitor because its context-influence signal is close in spirit to what we measure. It reports 0.85 AUROC for unfaithfulness detection on RAGTruth and biographies, a 12% faithfulness gain over greedy decoding, and 19% over context-aware decoding. We run it on the replayed documents with its published thresholds retuned on our held-out split. It serves both as a detector, where we ask whether its score predicts non-following under counterfactual edits, and as a generator.

RHIO-style tuning. Improving Contextual Faithfulness of Large Language Models via Retrieval Heads-Induced Optimization trains against unfaithful samples produced by masking attention heads that copy context, and reports 83.77% FaithScore on GroundBench for its 13B model. We apply its training recipe at matched model size and matched fine-tuning tokens. Its data guarantee that context contains the answer, so its behaviour under confident, conflicting priors is exactly what is untested.

Untrained extract-then-answer. This is the decisive ablation. It uses the identical two-step pipeline with the grounded-note check, but only prompting, no training. It isolates whether the gain comes from the bottleneck structure or from training. Further ablations keep the documents visible to the answer step (no bottleneck), drop the question from the answer step, and remove the span-grounding penalty.

What counts as beating the baselines. Our method beats them if, at equal or better overall accuracy, it raises the evidence-following rate on high-confidence override edits by at least 15 points over the strongest of SYNCHECK/FOD and RHIO. It must also not raise capture of subtly wrong evidence on the "toward" edits. It must hold this margin over the untrained pipeline on at least two of three domains.

Why not just…?

Why not just use a bigger frontier model? Scale does not remove the behaviour we target. The 57% post-hoc citation rate comes from a 104B model built for retrieval (Correctness is not Faithfulness in Retrieval Augmented Generation Attributions). Larger models also have more confident priors on popular financial and legal facts, which our hypothesis predicts makes override worse. We include at least one frontier API model in the measurement study, using the sampling-based confidence measure, so this becomes an empirical result rather than an assumption.

Why not just turn reasoning off for lookups? This could remove the T1 penalty. It is a real competitor on T1 and we report it as a baseline. It cannot be the answer for T3 or TheoremQA, where multi-step synthesis is needed. The SciAux numbers do not settle whether thinking helps under misleading context. The file reports 77.03% for thinking on and 74.87% for thinking off under two misleading snippets, yet describes thinking on as the worse of the two. We therefore rerun that condition ourselves before relying on it.

Why not just prompt for grounding, or use an untrained extract-then-answer pipeline? This is the most serious threat. The executor-ignores-author result in Unveiling Confirmation Bias in Chain-of-Thought Reasoning suggests an untrained note writer will simply write its prior into the note. The grounding check would then fail visibly rather than silently, which is already worth having. If the untrained pipeline matches the trained one, we will report the bottleneck structure plus the free span check as the contribution and drop the training claim.

Why not just apply SYNCHECK as is? Its context-influence KL tells us whether context changed the output distribution at all. It does not tell us whether the specific retrieved value determined the answer. It was also validated only on static corpora. Our detector comparison measures directly whether its score predicts counterfactual non-following on replayed domain sessions. If it reaches high AUROC there, the measurement half of our proposal becomes a cheap monitoring recipe, and the intervention must justify itself on generation quality alone.

Proposed Experiments and Decisive Checks

We will run open-weight reasoning models with a thinking toggle, including Qwen3-8B, which SciAux already reports on, at 7–14B. We add one frontier API model for black-box measurement only. The first decisive check is the confidence-binned evidence-following curve on FinSearchComp override edits. A flat curve refutes the core claim. The second is whether the thinking-on minus thinking-off accuracy gap on T1 concentrates in the high-confidence, prior-conflicting bin. The third, on TheoremQA theorem edits, separates "ignored the theorem," where the answer matches the unedited gold, from "applied it wrongly," where the answer matches neither. This settles the open theorem-augmentation question. The fourth compares trained against untrained bottlenecks on all three domains at matched compute, reporting accuracy, evidence-following, capture rate on wrong-evidence edits, and the fraction of notes passing the span check.

Human legal and financial annotators validate a random 200 edits per domain for plausibility, so that low following cannot be blamed on edits the model rightly distrusts as absurd.