Video models cannot afford to look at every frame at full resolution, so they choose what to look at. Current methods choose clips or image patches that resemble the question, or that resemble the model's current guess at the answer. Text-only studies show that language models confirm instead of testing. They pick tests that fit their current hypothesis (Failing to Falsify), and they write step-by-step reasoning that defends an answer they had already committed to (Unveiling Confirmation Bias in Chain-of-Thought Reasoning). Put these together and evidence selection driven by the model's own guess becomes a mechanical form of confirmation bias. The footage that would show the guess is wrong is the footage least similar to the guess, so it is the first to be discarded. The damage is worst in multi-hop questions, where the clue for a later step does not resemble the original question.
We propose to measure this "confirmatory looking" in video and multi-image reasoning agents, and to replace relevance-based selection with a rule that spends a fixed visual budget on evidence that separates competing answers. The rule scores each candidate clip by how differently it matches the rival hypotheses. One temperature knob controls how much weight goes to the leading guess, so the rule moves continuously between looking that confirms and looking that refutes. The method needs no training and can be added to existing adaptive frame-selection loops. The question we test is narrow: at a matched token budget, does choosing evidence that tells hypotheses apart beat choosing evidence that is relevant to the question or to the current guess?
Core Claim
When an agent picks frames or patches by similarity to its current hypothesis, it has a measurable bias toward confirming footage. This bias explains a meaningful share of its errors, specifically the cases where a confidently wrong first guess survives to the final answer. Selecting evidence by the disagreement it creates among the top answer candidates, with candidate weights deliberately flattened, should reduce these lock-in errors at equal compute. It should beat three simpler fixes: a debiasing prompt, a random exploration reserve, and majority voting over several runs.
Research Program
The setting is long-video and multi-image question answering with a fixed visual token budget. Each of the four input directions plays a distinct role here. Video and multi-image reasoning tasks that need several hops, as in Video-Holmes and MMhops-R1, are the application regime. The visual token budget is a hard constraint that forces selection in the first place. Confirmation bias is the failure mechanism. Contextual distraction, the drop in accuracy when irrelevant material fills the context, becomes a controlled stress test: we plant a misleading early clip and watch how the model handles it.
The work has three parts. The first is a diagnostic. Failing to Falsify tracked the ratio of hypothesis-incompatible to hypothesis-compatible tests in the Wason 2-4-6 task, where a model probes a hidden number rule. We build a video analogue of that ratio. At each selection step, we record the model's leading answer and score every clip it chose for whether it fits the leading answer better or worse than the strongest rival. We use an open model such as Qwen2.5-VL-7B, which exposes token log-probabilities, and read belief off the answer distribution over multiple-choice options. This follows how Unveiling Confirmation Bias used the entropy of the direct answer as a proxy for belief strength. We also record the lock-in rate: the fraction of questions where a wrong first answer is still the final answer after evidence gathering.
The second part is the selection rule. For each of the top $K$ answer candidates $h_k$, the model writes one sentence predicting what would be visible if $h_k$ were true. Each candidate clip $s$, represented by a cheap low-resolution preview in the spirit of the low-resolution summary frames in DynFocus, is then scored as
$$D(s) = \sum_{k=1}^{K} b_k^{(\tau)} \Big( m(s,h_k) - \sum_{j=1}^{K} b_j^{(\tau)} m(s,h_j) \Big)^2,$$
where $m(s,h_k)$ is the similarity between clip $s$ and the predicted observation for $h_k$, and $b_k^{(\tau)} \propto p(h_k)^{1/\tau}$ is the model's belief in $h_k$ after tempering with temperature $\tau$. The score is a belief-weighted variance. It is high for clips that match some hypotheses and not others, and it is zero for clips that fit every hypothesis equally.
The temperature is what makes the bias explicit. As $\tau \to 0$, all weight falls on the leading guess, the variance collapses, and in practice selection reduces to similarity with that guess, which is confirmatory looking. As $\tau$ grows, rival answers keep influence even when the model is confident, and the rule seeks clips that would separate them. Full-resolution tokens are spent on the top-scoring clips. Beliefs are then recomputed and the loop repeats under a fixed total budget. We implement the rule as a replacement for the scoring step inside the iterative loop of A.I.R., which picks frames from query-frame similarity and then densely resamples around confirmed frames. Everything else in that loop stays the same, so any difference comes from the selection criterion alone.
The third part is a perturbation benchmark that tests resistance to distraction. Starting from Video-Holmes items, we insert a short decoy clip early in the video that supports a wrong option. We take the decoy from another film, choosing one whose predicted-observation similarity favors that wrong option. This is a visual counterpart to the narrative distractors that mislead text models (Will the Real Linda Please Stand up; Less is More: Summary of Long Instructions). It lets us cause confident wrong first beliefs on purpose instead of waiting for them to occur.
This program also gives a concrete test of a tension in the literature. Reasoning traces reduce confirmation bias in Wason-style tasks (Failing to Falsify), yet turning on "thinking mode" makes video models hallucinate more (ReWatch-R1). One explanation is that the Wason task returns clean yes/no feedback on every test, whereas a video agent's own looking returns evidence that is biased toward its current guess. If that explanation is right, reasoning should help video models once the evidence they gather is chosen to discriminate between answers.
Baselines
The naive reference is uniform frame sampling at the same visual token budget. A.I.R. reports that its adaptive selection improves QwenVL-2.5 on NextQA by +7.0% absolute over uniform sampling. That gap is the scale against which our gain over A.I.R. should be read.
The strongest training-free competitor is A.I.R. itself, which uses query-similarity proposals, model-judged candidate frames, and dense local resampling. We run it with its published procedure on the same backbone, the same frame and token budget, and the same number of selection rounds. Beating it means higher accuracy at matched tokens and matched wall-clock time, concentrated on items with confident wrong first beliefs.
We also include one-shot query-similarity patch pruning with TRIM, which keeps about 123 of 576 image tokens, so that we cover selection driven only by the question.
The strongest trained system is Video-o3. It learns, through reinforcement learning, to call a cropping tool that zooms into short video segments. It reports 46.5% on Video-Holmes and 72.1% on MLVU. Where weights are available, we wrap its tool calls with our scoring rule, re-ranking the candidate segments it proposes, and compare at its 8-turn limit. Where they are not, we report its number only as a reference, without a matched-compute comparison. For reference, Video-Holmes reports Gemini-2.5-Pro at 45.0%. On multi-image questions, MMhops-R1 reports 51.35% on Bridging and 22.01% on Comparison. There, our rule re-ranks which retrieved images get high-resolution tokens.
The key ablation sweeps $\tau$ from near zero, which reproduces confirmatory looking, to large values, with all else fixed. A second ablation keeps the flattened rival hypotheses but scores clips by plain similarity to a rival's predicted observation instead of by variance. This separates "look for the alternative" from "look for what tells the alternatives apart."
Why not just…?
Why not just add a Think-in-Opposites prompt to the agent? This is the most direct transfer from Failing to Falsify. There, telling models to consider the opposite raised rule-discovery rates from 42% to 56%. It is a real threat, and we run it as a baseline on top of A.I.R. Our expectation that it falls short rests on Unveiling Confirmation Bias, which found that confident models ignore rationales that contradict their prior. A prompt changes what the model says, but the clips are still chosen by similarity to the original query. If the prompt matches our rule on lock-in rate and accuracy, the selection rule adds nothing, and we will report that.
Why not just reserve part of the budget for random clips? Random exploration also breaks the confirmatory loop and costs nothing to design. We match the reserve fraction to the share of budget our rule sends to non-leading-hypothesis clips. If random exploration recovers most of the gain, the claim about discriminative selection fails and the finding reduces to "spread your looking around." The perturbation benchmark is where the two should differ most: random clips only rarely land on the specific footage that rebuts the decoy.
Why not just run several trajectories and take a majority vote? Voting averages out sampling noise. Confirmation bias, however, is systematic. A strong decoy should pull most runs toward the same wrong answer, so voting should not reduce lock-in on decoy items. We give voting the same total token budget, split across runs, so the comparison is fair. If voting matches our rule on the decoy items, the bias is noise rather than structure, and our premise is wrong.
Proposed Experiments
The main runs use Video-Holmes, NextQA, MLVU, and MMhops at three token budgets. We report accuracy, the video incompatible-to-compatible ratio, lock-in rate, and latency.
The first decisive check is diagnostic. Under similarity-based selection, a lower incompatible-to-compatible ratio should predict failure, as it did for text in Failing to Falsify. This prediction fits the Video-Holmes finding that reasoning errors make up about 60% of failures and missed visual clues about 35%, because errors driven by lock-in would appear under the reasoning label. If the correlation does not show up, the premise fails and we stop.
The second check concerns where the gains fall. Improvements should come mainly from questions where the initial answer entropy was low and the initial answer was wrong, with no loss on questions the model initially got right. Accuracy should also rise as $\tau$ increases and level off past a moderate value.
The third check uses the decoy-inserted Video-Holmes set. Our rule should cut the lock-in rate relative to A.I.R., the opposites prompt, the random reserve, and majority voting, each at matched tokens.
A final check tests the explanation for the reasoning-mode tension. We turn reasoning traces on and off under confirmatory and discriminative selection. If our explanation holds, reasoning should hurt under confirmatory selection, as ReWatch-R1 observed, and help under discriminative selection.
References
- Failing to Falsify: Evaluating and Mitigating Confirmation Bias in Language Models
- Unveiling Confirmation Bias in Chain-of-Thought Reasoning
- Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?
- MMhops-R1: Multimodal Multi-hop Reasoning
- DynFocus: Dynamic Cooperative Network Empowers LLMs with Video Understanding
- A.I.R.: Enabling Adaptive, Iterative, and Reasoning-based Frame Selection For Video Question Answering
- Will the Real Linda Please Stand up...to Large Language Models? Examining the Representativeness Heuristic in LLMs
- Less is More: Summary of Long Instructions is Better for Program Synthesis
- ReWatch-R1: Boosting Complex Video Reasoning in Large Vision-Language Models through Agentic Data Synthesis
- Less is More: A Simple yet Effective Token Reduction Method for Efficient Multi-modal LLMs
- Video-o3: Native Interleaved Clue Seeking for Long Video Multi-Hop Reasoning