Does the Search Agent Believe the User or Its Own Hunch?

Conversational search systems ask the user clarifying questions ("was the jacket red or orange?") and narrow a pool of candidates with each answer. On person search, video search and product search, this loop roughly doubles retrieval accuracy over one-shot search. Every one of these systems is evaluated against a simulated user who always answers correctly. Real users misremember and misreport. Separately, work on chain-of-thought faithfulness shows that language models often settle on an answer early and then write reasoning that defends it. That kind of reasoning is called post-hoc rationalization: a plausible justification for a conclusion the model had already reached.

Put these two facts together and a specific failure becomes likely. An agent that has formed an early guess about the target may accept a wrong user answer when it fits that guess, and explain away a correct answer when it does not. Its accuracy then collapses under realistic user noise, and the collapse is lopsided: wrong answers that agree with the guess do far more damage than wrong answers that disagree with it. We propose to measure this lopsidedness, which we call confirmation asymmetry, with controlled answer-injection experiments. We also propose a narrow fix and test it honestly. The fix scores each candidate against each answer in isolation, without the agent's running hypothesis in view, and combines the evidence with an explicit model of how often users are wrong. The comparison that matters most is against a plain Bayesian tracker that uses no language-model reasoning at all.

Core Claim

In interactive retrieval, the language model's free-form reasoning over the dialogue history is where its early hunch leaks into how it interprets user answers. This leak should show up as a measurable asymmetry. Suppose the user gives a false answer that is consistent with the agent's current top candidate. That answer should push the true target down the ranking more than an equally false answer that contradicts the top candidate. We further predict the asymmetry is larger when the model's explicit thinking mode is on, by analogy with Thinking in a Crowd. That paper shows Qwen3-8B with thinking enabled falls from 96.22% to 77.03% under two misleading retrieved snippets, which is worse than the 74.87% it gets with thinking off. Its model built elaborate justifications for the bad premise. Our second claim is about where the fix belongs. If the scoring of each answer against each candidate is made blind to the agent's hypothesis, most of the damage should disappear, and no faithfulness training is needed.

Relation to Prior Work

The interactive-retrieval systems supply our testbeds, and they also share the gap we exploit. LLaVA-ReID trains a multimodal questioner that picks hard look-alike images and asks discriminative questions to a simulated witness. IVCR-200K pairs a fast retriever with a multimodal re-ranker over multi-turn video queries. ProductAgent grounds its clarifying questions in catalog statistics. All three assume the user answers truthfully. We keep their pipelines and change only the user.

Two papers motivate and partly constrain the design. PaMis is the closest existing work. It detects patient misreports in medical dialogue using entity graphs linked to a medical knowledge graph, and it reports misreporting in 15–20% of real clinical cases. PaMis treats misreporting as a property of the user's statements. It does not ask whether the model's own leanings decide which misreports get through, and that question is our focus. Thinking in a Crowd establishes that reasoning traces rationalize misleading context, but only in single-turn question answering with retrieved snippets. A dialogue differs because the misleading context is generated in response to the agent's own questions, and the agent holds a hypothesis while reading it.

The faithfulness literature supplies the mechanism and rules out some fixes. Unveiling Confirmation Bias in Chain-of-Thought Reasoning shows that a strong initial hunch, measured by low entropy over the direct answer, makes models override their own step-by-step logic. We use that entropy as a per-turn measure of hunch strength. How Likely Do LLMs with CoT Mimic Human Reasoning? and On the Hardness of Faithful Chain-of-Thought Reasoning find that larger and post-trained models are often less faithful, and that steering toward faithfulness costs accuracy, up to 20% on TruthfulQA. These findings argue against any fix that tries to make the reasoning itself faithful. Measuring and Mitigating Post-Hoc Rationalization in Reverse Chain-of-Thought Generation shows that telling a model not to anchor makes anchoring worse and degrades multi-turn dialogue. That result rules out the cheapest mitigation, so our fix is structural rather than instructional. Do Language Models Know When They're Hallucinating References? motivates the idea that independent, context-free queries expose inconsistencies that a single conditioned pass hides. We apply the same principle to scoring answers against candidates.

Research Program

The first component is a noisy-user simulator that supports controlled injection. In each dialogue round the simulator answers the agent's question correctly by default. On a fraction $\rho$ of rounds, where $\rho \in \{0, 0.1, 0.2\}$ to bracket the PaMis clinical rate, it gives a false answer. Each false answer is one of two kinds, and we know which kind it is because we read the agent's current top candidate before the simulator replies. A congruent false answer matches an attribute of the agent's current top candidate, which by construction is not the target. A neutral false answer is a wrong value chosen not to match the top candidate. A third condition gives "not sure" answers, to separate uncertainty from error.

From this we define confirmation asymmetry. Let $\Delta_c$ be the average drop in the target's final rank after five rounds caused by congruent false answers, and $\Delta_n$ the drop caused by neutral ones, both measured relative to the same dialogue with a truthful answer. Confirmation asymmetry is

$$A = \Delta_c - \Delta_n .$$

An agent that weighs evidence without regard to its own hypothesis has $A \approx 0$. Each false answer still carries some information about which candidates are being discussed, so we calibrate the expected value of $A$ with the Bayesian tracker described below. That tracker has no hunch, so its $A$ is the reference point.

The second component is a question-side audit. We plant a hypothesis by reordering the candidate list or adding a single-sentence prior guess to the context. We then measure how much the expected information gain of the agent's next question changes, computed over the current candidate pool. This tells us whether hunches also produce leading questions, not only lopsided interpretation of answers.

The third component is the mitigation, which we call hunch-blind scoring. The same language or vision-language model scores each (candidate, question, answer) triple in a fresh context. That context contains no dialogue history and no ranking. The output is a compatibility probability, which we combine across rounds with a symmetric noise model. Because every candidate is scored in an independent context, no single candidate's score can be shaped by the agent's current favorite. The agent's model still writes the questions, and an answer whose evidence conflicts with earlier rounds triggers one confirmation re-ask.

The key ablation uses the identical scorer with the running dialogue and the current top candidate made visible. This ablation isolates the claim that the leak runs through hypothesis-conditioned reasoning.

Baselines

The naive reference point is non-interactive retrieval on the initial description. On Interactive-PEDES it reaches 35.86% Rank@1 in LLaVA-ReID, and any interactive method under noise must stay well above it.

The strongest existing method is LLaVA-ReID itself, which reports 73.20% Rank@1 after five rounds with a truthful simulated witness. We rerun it under our noise conditions using the released questioner, the same five-round budget and the same simulator seeds. We find no reported result on Lacuna for LLaVA-ReID or any interactive retriever under user noise, so we report its noisy numbers ourselves.

The third baseline is a chain-of-thought agent built on the same backbone. It reads the full dialogue and re-ranks, and we run it with thinking on and off, in the Qwen3 style. This baseline is where we expect the asymmetry to be largest.

The fourth is a PaMis-style consistency checker. We adapt it by replacing the medical knowledge graph with the attribute graph of the candidate pool. It flags contradictions and asks clarifying questions in response.

The fifth baseline is the most threatening: a model-free Bayesian tracker. It selects questions by maximum information gain over the structured attributes that GPT-4o extracted for Interactive-PEDES and updates its beliefs with the same noise model our method uses.

Every method gets the same retriever, the same candidate pool size, the same five turns (re-asks count as turns) and the same tuning budget on a held-out split. Our method beats the field if it achieves near-zero asymmetry and the best Rank@1 at $\rho = 0.2$, by a margin larger than the seed-to-seed variance over three seeds, while staying within two points of LLaVA-ReID when $\rho = 0$. Against the Bayesian tracker, success means a clear advantage on IVCR-200K and on attributes not covered by the extracted metadata, where the tracker has nothing to score.

Why not just…?

Why not just tell the agent that users can be wrong? Instructing a model about what not to anchor on increases anchoring and harms multi-turn coherence, according to the reverse chain-of-thought study. Faithfulness steering through prompting gave marginal gains at real accuracy cost in Ley et al.. We still include a "users may misreport" prompt variant of the chain-of-thought agent, so that this claim is tested in our setting rather than assumed.

Why not just use a bigger model? Bao et al. find that scale alone does not strengthen the causal link from reasoning to answer, and that larger models become more convincing explainers of answers they had already chosen. We test this directly by measuring confirmation asymmetry across backbone sizes within one model family. If asymmetry shrinks with scale, our case for the mitigation weakens, and we will report that.

Why not just drop the language model from belief tracking and use a Bayesian tracker over attributes? This is a real threat. On Interactive-PEDES, where attributes are already decomposed, the tracker may match our method. If it does, the finding is still informative: the language model's reasoning should not touch belief updates at all. Our method's claimed advantage is limited to answers and attributes that fall outside fixed metadata, namely free-form video moments in IVCR-200K and open-ended witness descriptions. The experiments report results separately on covered and uncovered attributes so the reader can see where any advantage comes from.

Why not just re-ask every question? Blanket re-asking halves the number of informative questions the agent can ask. Because all methods share a fixed turn budget, this baseline pays that cost openly. We include it as a variant to show whether targeted re-asks beat uniform ones.

Proposed Experiments

The primary testbed is Interactive-PEDES with five rounds. The secondary testbed is IVCR-200K, where R@1 rises from 38.4% on the first turn to 87.5% by the seventh in IVCR-200K. On IVCR-200K we test whether this gain from conversation survives noise. PROCLARE serves as a structured product-search check, though we found no absolute numbers for it on Lacuna. MedDG provides real misreport annotations, which we use to confirm that our synthetic congruent and neutral answers resemble actual patient errors. PaMis's detection F1 of 61.44 against GPT-4's 59.28 serves as the detection reference. Metrics are Rank@1 and R@1 at each noise level, confirmation asymmetry $A$, the shift in information gain on the question side, and the correlation between per-turn hunch entropy and susceptibility to congruent false answers.

The decisive checks are three. First, $A$ must be significantly positive for the chain-of-thought agent and larger with thinking on. If it is near zero, the phenomenon is not real and the project becomes a robustness benchmark. Second, the hypothesis-visible ablation must reintroduce most of the asymmetry that hunch-blind scoring removes, which would locate the leak in hypothesis-conditioned reasoning. Third, on attributes outside the structured metadata, hunch-blind scoring must beat the Bayesian tracker. If it does not, we recommend the tracker.