Recent video benchmarks paint a stark picture. Video-language models, which answer questions about video clips, mostly fail when the answer depends on how things change over time rather than on what any single frame shows. On TimeBlind, built from pairs of videos with the same scenes but different temporal structure, the best model scores 48.2% against 98.2% for humans. On Minimal Video Pairs the best open model scores 40.2%. On TimeLogic's yes/no questions, models score about 50–54%, close to chance. These numbers, however, come from evaluation settings that differ from model to model. Each model is fed a different number of frames, chosen by a different sampling rule, sometimes with timestamps and sometimes without, and prompted in different ways. Work on benchmarking in other subfields keeps showing that settings like these can account for much of the gap between methods: in point clouds, word embeddings, and multimodal recommendation, freezing the training and evaluation recipe changed which method came out ahead.
This note proposes an audit of video-language evaluation in which the model is the only thing that varies. We would run a fixed grid of evaluation protocols across a set of models and shortcut-resistant temporal benchmarks. The grid covers frame budget, sampling rule, timestamp exposure, and prompting style. Under every protocol, each model also sees the same clip with its frames shuffled or reversed. The audit then answers three questions. How much of the variation in scores comes from the model, and how much from the protocol? Do model rankings survive reasonable changes of protocol? And, under its own best protocol, does each model actually use frame order? The last question tests whether "temporal blindness" belongs to the architecture or to the measurement.
Core Claim
The claim to test is that current temporal leaderboards mix up two things. The first is whether a model uses temporal order. The second is how well the default evaluation settings happen to suit that model. The two can be separated with one quantity, measured under identical conditions:
$$\Delta_{m,p} = \text{Acc}_{m,p}(\text{ordered}) - \text{Acc}_{m,p}(\text{shuffled}),$$
Here $m$ is a model, $p$ is a protocol (frame count, sampling rule, timestamps, prompt), and Acc is pair-level accuracy, meaning credit is given only when both members of a minimal pair are answered correctly. A model that relies on single frames has $\Delta \approx 0$ under every protocol. A model that has temporal ability but is starved by its default protocol has small $\Delta$ under the default and large $\Delta$ under some other protocol. We predict two things. Rankings of models by raw accuracy will be noticeably unstable across the grid. Rankings by $\max_p \Delta_{m,p}$ will be more stable and will separate video-finetuned models from image-only ones and caption-based cascades. If neither prediction holds, then temporal blindness is architectural and the existing diagnostic benchmarks are already measuring it faithfully. That would also be a useful result.
Relation to Prior Work
The motivating evidence comes from the shortcut-resistant benchmarks. TimeBlind pairs videos that share static content and scores both pairs jointly. It reports that GPT-5 drops from 46.3% to 4.5% with a single frame and to 3.0% with shuffled frames, and that going from 8 to 32 frames gives negligible gains. MVP builds 27,414 physically grounded minimal pairs and filters out questions solvable from one frame. ViLMA combines a static proficiency test with temporal counterfactual captions. TimeLogic generates formal temporal-logic questions and uses blank-frame controls. EgoSchema requires long temporal evidence and notes that more frames sometimes hurt. All five fix one evaluation setting per model, usually the model's default, and report a single number.
Our audit differs from them in three ways. It treats the protocol as a controlled factor. It applies the ordered/shuffled/reversed controls inside every cell of the grid rather than once at the aggregate level. It compares models on the same footing. TimeBlind's frame-scaling result is the most direct prior evidence. It constrains our hypothesis, but it does not settle it, because it varied only the number of frames for frontier closed models, not sampling rule, timestamps, or prompting, and it did not measure order use separately from accuracy.
Two other papers supply the methods we reuse. From VHELM and AHELM we borrow standardized prompts, fixed decoding (temperature 0, capped output length), and the practice of including modular baselines. AHELM's finding that a transcription-plus-LLM pipeline ranked 6th of 17 systems and held 3 of the top 5 robustness slots is the reason we treat a caption cascade as a serious competitor. Video Finetuning Improves Reasoning Between Frames supplies Visual Chain-of-Thought (vCoT), a two-step prompt that first describes what consecutive frames share and then infers the event between them. It gave a 7.4% gain to an image-only model on EgoSchema but only 1.6% to its video-finetuned counterpart. That asymmetry is exactly the kind of protocol-by-model interaction that can reorder a leaderboard. The broader concern that hidden implementation details masquerade as method differences is established by Jurgens et al., Pomo et al., and the point-cloud baseline study. We carry that lens over to temporal inputs.
Research Program
We would build an evaluation harness that wraps each model behind a single interface. The interface takes a clip, a frame budget, a sampling rule, a timestamp flag, a prompt template, and a frame-order transform, and returns an answer under fixed decoding.
The protocol grid has four parts. Frame budgets are 1, 4, 8, 16, 32 and, where context allows, 64. Sampling rules are uniform, segment-random, and change-based keyframes, where frames are chosen at large pixel differences. Timestamps are either rendered into the prompt or omitted. Prompting is direct answer, generic step-by-step, or vCoT. Frame-order transforms are identity, shuffle, and reverse, plus a single-frame condition.
Cells that a model cannot accept, such as 64 frames on an 8-frame model, are marked missing rather than adapted, and the analysis handles missing cells explicitly. Every model gets the same compute envelope per item. Prompts are written once and never tuned per model.
For each model we report three quantities. The first is default-protocol accuracy, which reproduces published numbers and checks the harness. The second is accuracy at its best protocol, chosen on a held-out tuning split and then frozen. The third is $\max_p \Delta_{m,p}$, also chosen on the tuning split. Across models we fit a simple variance decomposition of score into model, protocol, and model-by-protocol terms, and we report Kendall's rank correlation between leaderboards produced by different protocols.
Baselines
The naive reference point is the leaderboard as published: each model evaluated with its default frames and a direct prompt. This is what the field currently uses. On TimeBlind the strongest reported result is Gemini 3 Pro at 48.2% Instance Accuracy, with GPT-5 at 46.3% (TimeBlind). On MVP the strongest is InternVL2.5-8B at 40.2%, ahead of GPT-4o at 32.5% (MVP). On ViLMA, InternVideo scores a combined 54.2%. On EgoSchema, the original paper reports models between 20% and 32% (EgoSchema). The audit beats this baseline if re-running under controlled protocols changes the published model ordering. A Kendall correlation below 0.6 between the default leaderboard and the best-protocol leaderboard, on at least two benchmarks, would count as meaningful reordering.
The strongest existing method for improving temporal answers without retraining is vCoT prompting (Video Finetuning Improves Reasoning Between Frames). We know of no reported vCoT result on TimeBlind, MVP, ViLMA or TimeLogic; the file contains only its EgoSchema gains. We run vCoT inside the grid with the same frames and decoding as the direct prompt, so any gain is attributable to the prompt alone.
The strongest system competitor is a modular cascade, modeled on the Whisper-plus-LLM baselines in AHELM and the caption-then-LLM setup in VisIT-Bench, where instruction-conditioned captions lifted GPT-4 from 31% to 91% success. In our cascade, a frozen image captioner describes each sampled frame, and the captions, with timestamps, go to a text-only language model. It uses the same frame budgets as the end-to-end models. Video models beat it only if their $\max_p \Delta$ exceeds the cascade's, not merely if their raw accuracy does. We also include image-only models, as in ViLMA's CLIP and BLIP-2 comparisons, as the "no temporal training" floor.
The ablation that isolates our key ingredient replaces the per-cell shuffled control with the single aggregate shuffle test used by TimeBlind. If the conclusions do not change, the per-cell control adds nothing and the audit reduces to a protocol sweep.
Why not just…?
Why not just let each model use its best settings and report the maximum? This "best-of-protocol" leaderboard is the most serious simpler alternative, and we compute it. Its weakness is that it rewards whatever exploits the remaining shortcuts. A model can gain accuracy from more frames because more static cues become visible, not because it uses order. TimeBlind's single-frame and shuffled numbers show that such cues exist even on minimal pairs. If best-of-protocol accuracy and $\max_p \Delta$ produce the same ranking, the simpler leaderboard suffices, and we will say so.
Why not just add more frames, since that is the obvious fix? TimeBlind reports negligible gains from 8 to 32 frames, and EgoSchema reports decreases for some models. Humans on EgoSchema still reach about 67% at 1 frame per second. Frame count alone is therefore unlikely to be the answer. Our grid tests it jointly with sampling rule and timestamps, which neither paper varied.
Why not just use a caption cascade and skip native video models? AHELM's results make this a real threat. If the timestamped cascade matches the best video model on $\Delta$ across TimeBlind and MVP, the honest conclusion is that current video encoders contribute no temporal signal beyond ordered text descriptions. ViLMA's near-chance action counting for image-language models suggests cascades will fail on fine-grained events. The cascade arm settles this directly.
Why not just adopt vCoT as the standard prompt? vCoT's gain differs between image-only and video-finetuned models: 7.4 points versus 1.6 on EgoSchema. Fixing it as the standard would systematically favor some architectures. We measure that interaction rather than assume it away.
Proposed Experiments and Decisive Checks
The benchmarks are TimeBlind (all 2,400 video-question pairs), a stratified 5,000-pair subsample of MVP spread across its three sources, ViLMA's five phenomena, TLQA-S, and the public EgoSchema subset. Models span the image-only versus video-finetuned contrast (LLaVA-NeXT and LLaVA-Video), older sparse-frame video models (VideoLLaVA at 8 frames, SeViLA at 32, InternVideo), InternVL2.5-8B, and GPT-4o and Gemini through their APIs, plus the cascade. Each benchmark is split 20/80 into a tuning portion and a reporting portion. All reported numbers come from the 80% portion, with item-level bootstrap intervals.
The first check confirms the harness: default-protocol runs must land within bootstrap error of the published figures above before any further analysis. The second decides whether protocol matters. If the model-by-protocol interaction explains at least as much variance as the model main effect on two or more benchmarks, current leaderboards are protocol-confounded. If it explains under 10%, they are not, and TimeBlind's architectural reading stands. The third decides whether temporal blindness is real. If every model's $\max_p \Delta$ on TimeBlind and MVP is indistinguishable from zero, the blindness is architectural. If the video-finetuned models show $\Delta$ well above image-only models and the cascade under some protocol, the deficit is partly a measurement artifact, and the winning protocol becomes a recommended reporting standard. The fourth applies EgoSchema's non-monotonicity as a sanity check: we expect the frame-count curves to differ by sampling rule. If they do not, sampling rule can be dropped from future standards.
The final deliverable is a released harness, the full results grid, and a recommended minimal reporting card for temporal benchmarks. The card would require a fixed frame budget, the sampling rule, the timestamp setting, the prompt, and the ordered-minus-shuffled score.
References
- TimeBlind: A Spatio-Temporal Compositionality Benchmark for Video LLMs
- A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairs
- ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Models
- TimeLogic: A Temporal Logic Benchmark for Video QA
- EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding
- VHELM: A Holistic Evaluation of Vision Language Models
- AHELM: A Holistic Evaluation of Audio-Language Models
- Video Finetuning Improves Reasoning Between Frames
- The S-Space Package: An Open Source Package for Word Space Models
- Disentangling the Performance Puzzle of Multimodal-aware Recommender Systems
- Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline
- VisIT-Bench: A Dynamic Benchmark for Evaluating Instruction-Following Vision-and-Language Models