What Happens When: Learning Temporal Orders of Events in Videos

Daechul Ahn, Yura Choi, Hyeonbeom Choi, Seongwon Cho, San Kim, Jonghyun Choi

TL;DR

This paper reveals a surprising flaw in current Video Large Multimodal Models (VLMMs): they can pass existing temporal benchmarks even when video frames are completely scrambled, relying on common-sense priors rather than true visual understanding. To address this, the authors introduce VECTOR, a new diagnostic benchmark that forces models to track arbitrary event orders, and MECOT, a fine-tuning and inference strategy that significantly improves genuine temporal reasoning.

What Problem Does This Paper Address?

While VLMMs excel at general video understanding, their ability to track the exact chronological order of multiple events has been vastly overstated. The authors discovered that state-of-the-art models perform remarkably well on existing temporal benchmarks even when video frames are randomly shuffled.

This happens because models exploit a "prior-knowledge shortcut." For example, if a video shows a campfire being built, a model doesn't need to look at the timeline to know that gathering stones happens before lighting the fire, and lighting the fire happens before ashes appear. If the frames are swapped, the model will often stubbornly predict the logical sequence rather than what is actually shown visually.

Figure 2. Biased prediction on event-ordering task. The model correctly predicts the event order [‘A’, ‘B’, ‘C’] for the original video (top). However, when events A and C are swapped (bottom), it maintains the same prediction, revealing its reliance on plausible scenario over visual temporal information.

To achieve true spatiotemporal understanding, VLMMs must be evaluated on tasks where common-sense expectations are broken, forcing them to rely strictly on visual evidence.

Methods

The authors tackle this problem from two angles: a new evaluation framework and a novel training/inference methodology.

1. The VECTOR Benchmark

To eliminate the prior-knowledge shortcut, the authors created synthetic videos by concatenating completely unrelated short action clips. Because the sequence of events is entirely arbitrary (e.g., playing guitar $\rightarrow$ stretching $\rightarrow$ eating ice cream), the model cannot rely on common-sense scripts to guess the order.

2. MECOT (Multi-Event instruction fine-tuning with Chain-of-Thought)

To help models learn true temporal coherence, the authors proposed a two-stage approach:

  • Instruction Fine-Tuning (IFT): They generated detailed, segment-wise descriptions of multi-event videos using a 7B VLMM and smoothed them into coherent narratives using GPT-4o-mini. They then fine-tuned a base LLaVA-OneVision model on 120k of these temporally enriched descriptions.
  • Chain-of-Thought (CoT) Inference: At test time, instead of answering a prompt directly, the model is guided to first generate a chronological summary of the video's events before outputting the final answer.

Datasets

  • VECTOR (New Benchmark): Contains 4.8k QA pairs and 16.8k action videos compiled from the validation split of Kinetics-700. It features 5 distinct sub-tasks split into event-level tracking (sequencing, relative sequencing, position identification) and pattern-level tracking (finding semantic outliers or pattern-breaking actions).
  • Prior-Knowledge Diagnosis: The authors used MECD and HiREST to quantify the prior-knowledge bias using a new metric called the biased ratio ($\eta$).
  • Generalization Benchmarks: The final model was evaluated across 9 established benchmarks, including EgoSchema, MVBench, LongVideoBench, VideoVista, MLVU, TVBench, TemporalBench, TempCompass, and VidComposition.

Key Insights and Ideas

  • The Shuffling Illusion: On many popular video benchmarks, open-source models retain over 82% of their original accuracy when frames are shuffled. This proves these datasets are testing static feature recognition and world knowledge rather than temporal reasoning.
  • VECTOR Exposes the Flaw: When evaluated on VECTOR, standard open-source VLMMs drop to near-chance performance, especially as the number of events increases from 4 (L1 difficulty) to 8 (L2 difficulty).
  • More Frames $\neq$ Better Tracking: Simply feeding more frames into a standard VLMM does not improve its long-context temporal reasoning. In fact, adding frames beyond a certain threshold can sometimes degrade performance. Models must explicitly learn to model event transitions.
Figure 1. Limitations of existing video benchmarks and our VECTOR benchmark. (a) Normalized accuracy (Acc.) indi- cates the ratio of a model’s accuracy on frame-shuffled videos to its accuracy on original videos. SoTA open-source VLMMs [15] (7B and 72B) achieve high accuracy on existing benchmarks but not on our proposed VECTOR. (b) We think existing benchmarks often allow models to succeed via prior knowledge of typical event scenarios, circumventing true temporal understanding presented in video. (c) In contrast, VECTOR explicitly evaluates event-order understanding independent of prior knowledge.

Key Results

  • Benchmark Defeat: On the VECTOR Event Sequencing task (L1), the baseline LLaVA-OneVision model scored 23.00% Exact Match (EM). MECOT boosted this performance to 41.67%.
  • Pattern Recognition: In Task 4 (identifying a semantic outlier in a sequence), MECOT achieved 38.67% (L1) and 35.67% (L2), outperforming all other open-source 7B baselines.
  • Downstream Generalization: MECOT doesn't just excel on synthetic data; it also improved performance on real-world benchmarks, raising EgoSchema accuracy from 60.10% to 62.25% and TempCompass from 64.11% to 66.53%.

Limitations and Future Work

Despite the clear improvements introduced by MECOT, a massive gap remains between AI and human performance. On the VECTOR benchmark, human annotators easily score between 80% and 100% across all tasks, whereas the best open-source models still hover below 50% on the easiest settings.

Furthermore, proprietary models like GPT-4o and Gemini 1.5 Pro, while superior to open-source models, also experience severe performance degradation when sequences scale up to 8 or more events. Future work will need to explore architectural changes in VLMMs to inherently capture sequential dynamics without relying on text-based chain-of-thought crutches.

Summary updated

Abstract

Video Large Multimodal Models (VLMMs) have shown impressive performance in video understanding, yet their ability to accurately capture the temporal order of multiple events remains underexplored. We interestingly observe that, even when video frames are scrambled, models perform very well on the existing benchmarks by comprehensive experiments. This implies that VLMMs may not necessarily rely on accurate sequential processing of visual events, but instead depend on prior knowledge of typical scenarios to answer the question. To benchmark temporal understanding capabilities in VLMMs, we propose VECTOR, designed to explicitly assess a model's ability to identify the temporal order of events. On this benchmark, we observe that various VLMMs often fail to understand the orders of events. To address this, we propose MECOT (Multi-Event instruction fine-tuning with Chain-of-Thought), which (1) trains models on detailed, event-by-event video descriptions and (2) using chain-of-thought prompts at inference to enhance temporal awareness. MECOT outperforms prior arts on VECTOR as well as improving performance on existing video benchmarks, implying effectiveness of temporal understanding. We release our code, model and datasets.
BibTeX
@inproceedings{ahn2026what,
  author = {Daechul Ahn and Yura Choi and Hyeonbeom Choi and Seongwon Cho and San Kim and Jonghyun Choi},
  title = {{What Happens When: Learning Temporal Orders of Events in Videos}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2026},
  doi = {10.1109/wacv61042.2026.00273},
}
Figure