Long videos are too long for a model to read every frame, so a newer class of systems navigates instead. A video agent reads short text summaries of segments, decides where to jump, opens a few clips, and answers from them. These agents are scored by answer accuracy alone, and accuracy cannot tell an agent that answered because of the clip it opened from one that would have given the same answer from any clip, or from none. This note proposes a test at the moment the agent opens a clip. We quietly replace that clip with a different one from the same video that has a near-identical summary but does not contain the answer, and we check whether the agent's answer changes. An agent that is really using what it sees should lose confidence or change its answer. An agent leaning on the wording of the question, on answer-option priors, or on the text summary should not notice the swap.
The idea moves a well-tested tool from document translation into video. That tool measures how much a model's prediction depends on real context compared with substituted context. We use it to restore the measuring range of long-video benchmarks whose headline accuracies have stopped separating real perception from shortcuts. If it works, the outcome is a per-question "evidence-locked" score for any navigating agent, plus a diagnosis of whether its decisions follow the pixels or the captions. Building this needs no new human annotation beyond the timestamps that existing benchmarks already provide.
Core Claim
Accuracy on long-video question answering overstates how much navigating agents rely on the evidence they retrieve. A counterfactual test that swaps the retrieved evidence, using hard look-alike distractors drawn from the same video, will expose this gap per question, re-rank systems that accuracy currently orders differently, and separate reliance on visual content from reliance on intermediate text.
The claim rests on three pieces of prior evidence that motivate it. In document-level translation, On Measuring Context Utilization in Document-Level MT Systems compares predictions under true versus shuffled context using conditional cross-mutual information (CXMI), a measure of how much the real context lowers the model's uncertainty about the correct output. With it, the authors showed that high task accuracy can coexist with near-total context neglect. Multi-encoder models gave only 0.07% of decision credit to the antecedent, versus 2.86% for simple concatenation, and often had negative CXMI. The file notes these utilization metrics have been studied almost only in translation. In video, A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairs showed that requiring correct answers on both members of a hand-built pair of similar videos dropped models from about 45% to about 27%. Single-video accuracy was therefore heavily inflated. Video-MME found that subtitles raise Gemini 1.5 Pro from 75.0% to 81.3%, which suggests that scores partly track text retrieval.
What these papers leave open is the agent setting. LongVideo-R1 navigates a hierarchy of summaries down to 16-second clips using a captioning tool and a clip-answering tool. It reaches 50.0% on LVBench in about 10.5 rounds. It reports that the agent is often misled by visually similar segments, and it never checks whether the final answer depends on the clip it reached. That reported failure mode is exactly what our distractor construction targets.
Research Program
The core instrument is an evidence-swap harness that sits between an agent and its tools. When the agent asks to inspect a clip $E$, the harness can instead return a substitute $\tilde E$. We compute a per-question utilization score
$$U(q) = \log p_\theta(y^* \mid q, E) - \log p_\theta(y^* \mid q, \tilde E),$$
where $q$ is the question, $y^*$ the reference answer, $p_\theta$ the agent's probability of that answer, $E$ the evidence the agent actually chose, and $\tilde E$ the substitute. Averaging $U$ over a benchmark gives a navigation analogue of CXMI. For API models without log-probabilities, $p_\theta$ is estimated from answer frequencies over repeated samples.
We pair this with a binary "evidence-locked accuracy." A question is credited only if the agent answers correctly with $E$ and its probability of $y^*$ under $\tilde E$ falls to chance level or below. This borrows the both-must-hold scoring logic of minimal video pairs. The difference is that it requires no curated pairs, because the counterfactual is built from the agent's own navigation trace.
The substitute is chosen in three ways, and comparing them is the scientific content. A random clip from the same video gives an easy baseline. A blank or shuffled clip mirrors the shuffled-context control from translation and the shuffled-frame control in Video-R1. The main condition is a look-alike clip. It is the clip in the same video whose caption embedding is nearest to that of $E$, and it is constrained by the benchmark's annotated evidence timestamps to lie outside the interval that answers the question. Swapping look-alikes is the realistic threat, because it reproduces the confusion LongVideo-R1 reports.
A second, modality-splitting arm swaps the pixels and the caption separately. The agent receives either the true frames with the look-alike's caption, or the look-alike's frames with the true caption. If answers track the caption, the agent's reasoning runs through text rather than vision. This tests directly whether the shortcuts documented on charts by Plot Twist carry over to captioned video navigation. There, removing answer-bearing chart titles dropped accuracy from 79% to the 30% range. The test also connects to the perception–reasoning split reported in MATH-Vision and GeoSQA. We can ask whether an agent fails because it never reached the right clip, reached it but ignored it, or used it and still reasoned wrongly.
A smaller third arm asks whether utilization can be trained. We add an evidence-swap bonus to reinforcement-learning rewards for a small agent, in the spirit of T-GRPO's ordered-versus-shuffled bonus, and trace the accuracy-versus-utilization trade-off that the translation work predicts. This arm is secondary. The evaluation instrument stands on its own if training fails.
Baselines
The naive reference point is plain multiple-choice accuracy on the same questions, as reported by each benchmark. Every system is also run blind, with the question and options but no video, to show how much accuracy is reachable from priors. Our metric must say something that this pair does not.
The strongest existing navigating system is LongVideo-R1. Its paper reports 50.0% accuracy on LVBench with a Qwen-3-8B agent and 60.7% with a Qwen3-VL-32B backbone, and claims over 5% gains over VideoTree and VideoAgent. The file gives no citations for those two, and we will include them only if public code exists. We will run released LongVideo-R1 checkpoints if available, or else reproduce its two-tool loop on the same backbone. Each system gets the same videos and the same frame budget per clip. All conditions use the same tool-call budget, set to its reported average of 10.5 rounds.
For a non-navigating comparison we use uniform-sampling models at 16 frames: Video-R1-7B (37.1% on VSI-Bench in its paper) and the Find, Fix, Reason student (reported +22.33% on VSI-Bench over a Video-R1-SFT baseline). For these models the "retrieved evidence" is the sampled frame set, and the swap replaces frames inside the annotated evidence interval with look-alike frames. This lets us ask whether training on evidence patches produces evidence dependence.
The key ablation is random-clip versus look-alike-clip swapping. If both give the same utilization scores, the hard-distractor construction adds nothing and the method reduces to a cheap shuffle control.
The proposal beats the baselines if three things hold together. Evidence-locked accuracy re-orders at least one pair of systems relative to raw accuracy, with rank changes outside bootstrap confidence intervals. Look-alike swaps reveal significantly more neglect than random swaps. Utilization on the original benchmark predicts accuracy under a held-out distribution shift better than raw accuracy does. The shift is Video-MME long videos with subtitles removed, against the 14.3% long-video drop and the 81.3% versus 75.0% subtitle gap reported there.
Why not just…?
Why not just compare against a question-only (blind) baseline? Blind accuracy tells you what fraction of a benchmark is guessable on average. It does not tell you whether a given correct answer came from the clip the agent opened. An agent can beat its blind score by using generic video statistics, such as the overall topic of the video, while ignoring the specific clip it navigated to. We report blind accuracy for every system, and the experiments will show whether per-question $U(q)$ carries information beyond it. If the blind gap and mean $U$ rank systems identically, we will say so and the contribution narrows to the per-item and modality diagnostics.
Why not just measure temporal grounding accuracy? LongVideo-R1 already reports a 10.9% grounding lead, and grounding checks whether the agent reached the right place. Reaching the right clip and using it are different events, and the translation finding that accuracy and utilization trade off suggests they can come apart. We measure both on the same items and report the cell that grounding alone cannot see: right clip found, answer unchanged when that clip is swapped out. If that cell is nearly empty, grounding is a sufficient proxy and we will say so plainly.
Why not just build minimal video pairs, as MVP does? Minimal pairs are the gold standard, but they need expert curation and were built for short physical-reasoning clips, not hour-long narratives. Our swap is automatic, so it raises an artifact risk. A spliced clip might be detectably off-distribution, or a look-alike might happen to contain the answer. We address the first concern by drawing substitutes only from the same video at native resolution, so nothing is synthetic. We address the second by having annotators label 300 sampled swap items and reporting the rate at which the look-alike actually contains the answer. If that rate is high, the look-alike condition must be filtered more tightly before its numbers can be trusted.
Why not just use frame-shuffling, as in T-GRPO? Shuffling tests whether temporal order matters. It does not test whether the content of a specific retrieved segment matters, and a model can rely on unordered "lucky" frames. Shuffling is one of our substitute conditions, so the comparison is run directly.
Proposed Experiments
The evaluation suite is LVBench, Video-MME (short, medium, and long splits, with and without subtitles), and MLVU, matching LongVideo-R1's evaluation. CGBench supplies evidence timestamps for distractor filtering. CGBench was also LongVideo-R1's training source, so its results are reported separately as potentially contaminated. VSI-Bench and MMVU serve the uniform-sampling models.
The first decisive check is the look-alike gap. Evidence-locked accuracy under look-alike swaps should fall well below raw accuracy for navigating agents, more so than under random swaps. If the two swap conditions give indistinguishable scores, the central mechanism is wrong.
The second check is the modality split. For caption-driven agents we expect answers to follow the swapped caption more often than the swapped frames. If they follow the frames, the text-intermediary concern raised against LongVideo-R1 is refuted, which is also worth reporting.
The third check is predictive validity. Across systems, mean $U$ on Video-MME with subtitles should predict the drop to subtitle-free long videos better than raw accuracy predicts it.
The fourth check is the training arm. Adding the swap bonus should raise $U$ at a measurable accuracy cost or none, giving the first video-domain trace of the accuracy–utilization trade-off. All results are reported with bootstrap intervals over questions, given the replicability concerns in Prompt Engineering Techniques for Language Model Reasoning Lack Replicability. For the strongest baseline on evidence-locked accuracy itself, no reported result was found on Lacuna for LongVideo-R1 under any context-utilization metric.
References
- On Measuring Context Utilization in Document-Level MT Systems
- A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairs
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
- LongVideo-R1: Smart Navigation for Low-cost Long Video Understanding
- Video-R1: Reinforcing Video Reasoning in MLLMs
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level
- Find, Fix, Reason: Context Repair for Video Reasoning
- Prompt Engineering Techniques for Language Model Reasoning Lack Replicability