Many teams now score video-understanding models with another model: a multimodal "judge" that writes a short justification and then gives a score or picks the better of two answers. These judges are trained on ordinary, forward-playing video. Separately, there is strong evidence that video-language models often ignore frame order. They recognize what is happening from single frames and then fill in the familiar story from their language training, even when the video shows that story played backwards. This proposal asks whether the judges share that blind spot. If they do, a model that correctly says "the glass un-shatters" would lose to a model that confidently says "the glass shatters." Asking the judge to reason step by step could make this worse, not better, because the judge's own fluent text can drown out the frames.
We propose to measure this failure directly. We would build paired-candidate tests from time-reversed and reordered videos, in which the correct answer contradicts the usual script. We then test a simple, training-free correction we call order-contrastive judging. The judge's verdict on the real video is compared with its verdict on randomly shuffled copies of the same frames. Shuffled copies keep every object and scene but destroy order, so whatever survives the subtraction should come from temporal evidence rather than from static recognition or the language prior. The note also includes a probe that separates two explanations for any remaining failure: the judge never encoded the order, or it encoded the order and then overrode it.
Core Claim
Our claim has two parts. First, rationale-first video judges will systematically prefer script-consistent but wrong candidates on time-reversed and reordered clips. The effect should be larger for judges tuned for long reasoning than for instruct judges, mirroring the static-image "thinking paradox." Second, when frame order is linearly decodable from the judge's hidden states, contrasting the judge against shuffled-frame copies of its own input recovers much of the lost accuracy. It should do this without hurting forward-video judging and at a cost of a few extra judge passes per comparison rather than extra passes per generated token.
The first part connects three findings in the file that have never been tested together. Semantic Gravity shows that video models asked whether disorder increases in reversed slow-motion footage collapse from above 98% forward accuracy to 19.33% (GPT-4o) and 10.67% (Qwen3-VL-8B), while humans reach 99.33%. It also shows these errors are confident rather than uncertain. That study concerns generators answering a single binary question with no reasoning trace. Are VLMs Seeing or Just Saying? shows that in static images, long reasoning creates "textual inertia." Qwen3-VL-32B-Thinking drops 48.3% under a mid-reasoning image swap versus 17.9% for the instruct version, and the drop grows with scale. Breaking Down Video LLM Benchmarks shows that most video benchmark accuracy survives frame shuffling. These three papers motivate the proposal. The judges we would audit, VideoJudge and LLaVA-Critic, supply the method under test. Both assume a judge that writes a rubric or rationale first is a reliable verifier, and neither tests it on perturbed temporal input.
Research Program
The first component is a counter-script judging set. We would take the 300 forward/reversed pairs of the Observational Entropy Benchmark from Semantic Gravity, which contains no humans, so social cues cannot leak the answer. We would add the Direction and Event Order items of TempCompass, which include reversed and temporally concatenated clips. For each clip and question, we write two candidate responses matched in length and fluency. One is faithful to the order actually played. The other gives the familiar narrative completion. On forward clips the faithful answer and the script agree. On reversed clips they disagree. Matching length and fluency matters because Xie et al. show that judges scoring directly latch onto exactly those surface cues. To check that the effect is not specific to physics, we also take the Temporal subset of a VBenchComp-filtered benchmark such as NExT-QA or VideoMME. VBenchComp is the filtering procedure of Breaking Down Video LLM Benchmarks. On this subset we generate candidates that differ only in event order.
The second component is the audit. We run each judge in three modes: direct verdict, rationale-first verdict, and rationale-first verdict with a long-thinking model. The main measurement is the reversal gap, meaning pairwise accuracy on forward pairs minus pairwise accuracy on reversed pairs. We also measure order sensitivity, the fraction of verdicts that change when the judge's input frames are randomly shuffled. A judge that genuinely reads order should change its verdict often on Temporal items and rarely on items that the VBenchComp filter labels Semantic. Finally, we check rationale-verdict consistency. An external checker labels whether the rationale explicitly mentions the observed direction, which lets us see whether the rationale cites order evidence and then ignores it, or never cites it at all. This last measurement speaks to the open question in Direction 2 about whether evaluator reasoning actually determines the verdict.
The third component is order-contrastive judging. Let $p(a \succ b \mid V, q)$ be the judge's probability, read from its verdict token, that candidate $a$ beats candidate $b$ for question $q$ on video $V$. Let $\pi_k(V)$ for $k = 1..K$ be random frame permutations. The corrected preference is
$$\Delta(a,b) = \log p(a \succ b \mid V, q) - \frac{\lambda}{K}\sum_{k=1}^{K} \log p(a \succ b \mid \pi_k(V), q),$$
where $\lambda \ge 0$ sets the strength of the correction. The judge prefers $a$ when $\Delta > 0$. The design borrows the idea of subtracting a degraded-input pass from Visual Contrastive Decoding. It differs from that method in two ways. The degradation removes order rather than adding pixel noise. And the contrast is applied once per judgment rather than at every token. The second difference matters because that method's doubled per-token cost is a known drawback, and because noise pushes models further toward word-frequency priors (Pi et al.). In the rationale-first mode, the judge writes its rationale on the true video. We then read the verdict probability under each frame condition while keeping that rationale fixed. A variant regenerates the rationale under each condition.
The fourth component diagnoses where the order information is lost. Following the linear probes of Diagnosing Knowledge Conflict in Multimodal Long-Chain Reasoning, we train a linear probe at each layer to decode playback direction from hidden states at the frame tokens and at the verdict token. That paper found static conflict signals concentrated in layers 15–22 of 7B models. If direction is decodable at the frame tokens but not at the verdict, the judge saw the order and then overrode it. In that case order-contrastive judging should help, and probe-guided steering becomes a natural next step. If direction is not decodable anywhere, the vision encoder never captured order, and no inference-time correction can help. This separates the two failure sources that Direction 4 lists as hard to tell apart. It also gives a principled reason to expect the method to work on some judges and fail on others. Gemini 2.5 Pro and Qwen3-VL-32B sit near chance (51–55%) in Semantic Gravity, which may mean their encoders do not register order at all.
Baselines
The "do nothing" baseline is each judge used exactly as released: a direct verdict, then a rationale-first verdict, on the same candidate pairs, with the same frame budget and the same decoding temperature. The strongest existing system is VideoJudge at 7B. It writes an instance-specific rubric before judging and reports 98.6% pairwise accuracy on its standard benchmark and 93.67% on its hard human-verified subset (VJ-H). LLaVA-Critic is the second trained judge; its 72B model reports Pearson 0.754 agreement with GPT-4o. We found no reported result on Lacuna for any judge's pairwise accuracy on reversed or shuffled video. We therefore expect a large reversal gap, but this is our hypothesis, not a cited figure. These two judges are the right competitors because they are the tools people actually use for video evaluation, and their published accuracy is the reason people trust them. Every method, including ours, runs on the same frames, at the same resolution, with the same candidate pairs. Hyperparameters ($\lambda$ and $K$) are tuned only on a held-out tenth of the TempCompass pairs and then frozen for all other tests.
Three ablations isolate the key ingredient. In the first, the contrast input is a Gaussian-blurred copy of the frames instead of a shuffled copy, while $K$ stays the same. If the blurred contrast works as well as the shuffled one, the benefit comes from cancelling the language prior in general, not from removing order specifically. In the second, $\lambda = 0$ with $K$ extra samples averaged on the true video. This controls for the gain that simply ensembling more judge passes would give. In the third, the contrast copy is the reversed video rather than a shuffled one. Reversal flips the order signal instead of removing it, so we expect it to be noisier. Order-contrastive judging counts as beating the field if it closes at least half of the reversal gap of the best rationale-first judge, loses no more than one point on forward pairs and on the VJ-H subset, and beats both the blurred-contrast ablation and the ensemble ablation on reversed pairs.
Why not just…?
Why not just tell the judge to attend to frame order and ignore prior knowledge? We will run this instruction-only judge as a baseline. The existing evidence is unfavorable. In ContextFocus, prompting gave only slight gains in text conflicts, and the paper's effective steering vectors had to be built on top of instructions. In VISUALSWAP, appending a reflective "look again" phrase inside the same turn produced almost no extra attention to the image. If careful instructions alone close the reversal gap, the proposal reduces to an audit plus a prompt recommendation. We would report that outcome as such.
Why not just reset the judge with a new conversational turn, which VISUALSWAP found restores grounding? This is the most serious threat. Are VLMs Seeing or Just Saying? found that a user-turn boundary breaks textual inertia in static images. We will run a two-turn judge in which the rationale is produced in one turn and a fresh turn asks for a frame-by-frame order check before the verdict. If the two-turn judge matches order-contrastive judging on reversed pairs, the simpler fix wins. Our method would then matter only for judges whose failure the probe locates as a deep override that a new turn does not touch. We would state this outcome openly rather than argue around it. We also test the two together, because removing inertia and removing static priors may be complementary.
Why not just have a text-only model judge from frame captions? VideoJudge already reports that text-only judges given detailed video descriptions do much worse on temporal evaluation. Temporal Chain of Thought notes that captioning errors carry forward through every later stage. Captions of reversed slow-motion footage are especially likely to name the event in its familiar direction ("a glass shattering"), so the script would enter before any judging happens. We include a caption-then-judge baseline mainly to confirm this on the counter-script set.
Why not just filter evaluation sets with VBenchComp and keep the judges as they are? Filtering removes questions that can be answered without the video or without order. It does not change how a judge scores a candidate on the Temporal questions that remain, and that is exactly where we predict the bias appears. We will report judge accuracy separately on VBenchComp's Semantic and Temporal subsets, which tests this directly.
Proposed Experiments
The first decisive check is the audit on reversed pairs from the Observational Entropy Benchmark. We would test VideoJudge-7B, LLaVA-Critic, Qwen3-VL-8B and 32B in both their Instruct and Thinking versions, and GPT-4o as judges. The hypothesis is confirmed if reversed-pair accuracy falls well below forward-pair accuracy for every judge and the Thinking judges show the larger gap. It is refuted if rationale-first judging shrinks the gap relative to direct judging. That would mean the frame-by-frame reasoning that Temporal Chain of Thought uses for frame selection also protects judges, which would be a useful finding in its own right.
The second check is order sensitivity on the VBenchComp Temporal and Semantic subsets. A grounded judge should change its verdicts under shuffling on Temporal items and stay stable on Semantic items. A judge that is equally stable on both is not reading order at all.
The third check compares order-contrastive judging against the blurred-contrast, ensemble, two-turn, and instruction baselines under the success criteria given in the Baselines section. We would report results by layer-probe outcome. The prediction is that gains appear only in judges where playback direction is decodable at the frame tokens. If gains also appear in judges where direction is not decodable, the subtraction is doing something other than restoring order information, and our explanation is wrong.
The final check measures the practical cost of a biased judge. We would rank several generator models on the Direction and Event Order splits of TempCompass using each judge and compare those rankings to TempCompass's exact-match ground truth. If uncorrected judges reorder generators in favor of models that retrieve the familiar script, then using such judges as reward models, as LLaVA-Critic does for preference optimization, would actively train temporal blindness into the next generation of models.
References
- Semantic Gravity: When Parametric Memory Overpowers Visual Thermodynamics in Video-LLMs
- Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examination
- Breaking Down Video LLM Benchmarks: Knowledge, Spatial Perception, or True Temporal Understanding?
- VideoJudge: Bootstrapping Enables Scalable Supervision of MLLM-as-a-Judge for Video Understanding
- LLaVA-Critic: Learning to Evaluate Multimodal Models
- TempCompass: Do Video LLMs Really Understand Videos?
- An Empirical Analysis of Uncertainty in Large Language Model Evaluations
- Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding
- Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization
- Diagnosing Knowledge Conflict in Multimodal Long-Chain Reasoning
- ContextFocus: Activation Steering for Contextual Faithfulness in Large Language Models
- Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames