What Happens When: Learning Temporal Orders of Events in Videos

Daechul Ahn, Yura Choi, Hyeonbeom Choi, Seongwon Cho, San Kim, Jonghyun Choi

TL;DR

This paper reveals a critical flaw in current video understanding models: they often rely on learned "common sense" rather than actual temporal event order. To address this, the authors introduce VECTOR, a new benchmark specifically designed to test temporal understanding, and MECOT, a novel method that combines detailed event-based fine-tuning with chain-of-thought reasoning to significantly improve temporal event order comprehension.

What Problem Does This Paper Address?

Large Multimodal Models (VLMMs) have become incredibly adept at understanding video content. However, a crucial aspect of human comprehension – understanding the temporal order of events within a video – remains poorly understood and evaluated. The authors discovered that many state-of-the-art VLMMs perform surprisingly well even when video frames are scrambled. This suggests they aren't truly learning the sequence of events but are instead leveraging pre-existing knowledge about typical scenarios (e.g., knowing that a campfire is usually built before it's lit). This "prior knowledge bias" hinders genuine temporal understanding and can lead to incorrect reasoning when events deviate from the norm. Existing benchmarks, therefore, fail to adequately assess this critical capability.

Methods

The paper introduces two main contributions:

  • VECTOR Benchmark: This is a novel benchmark designed to explicitly test temporal order understanding, independent of prior knowledge. It achieves this by creating synthetic videos by concatenating unrelated action clips with abrupt transitions. This forces models to rely on the visual sequence itself rather than common-sense assumptions. VECTOR includes two main task groups: "Event ordering" (sequencing individual events) and "Pattern ordering" (identifying semantic patterns and anomalies).
  • MECOT (Multi-Event instruction fine-tuning with Chain-of-Thought): This is a novel method proposed to improve temporal understanding. It has two key stages:
  • Instruction Fine-tuning: Models are fine-tuned on detailed, event-by-event descriptions of videos. This helps them learn the nuances of temporal progression.
  • Chain-of-Thought (CoT) Inference: At inference time, MECOT uses CoT prompts to guide the model through a structured reasoning process, explicitly articulating the temporal relationships between events before answering questions. This enhances temporal awareness.

Datasets

  • Training: The paper uses existing datasets for creating detailed video descriptions for MECOT's instruction fine-tuning stage. Specific datasets are not explicitly listed for this purpose, but the process involves generating segment-wise descriptions and merging them.
  • Evaluation:
    • VECTOR: This is a newly introduced benchmark, comprising 4.8k QA pairs and 16.8k action videos across five sub-tasks. It's designed to be challenging and prevent shortcut-based guessing.
  • Existing Benchmarks: The paper also evaluates MECOT on several established video understanding benchmarks, including EgoSchema, MVBench, Long VideoBench, Video Vista, MLVU, TVBench, TemporalBench, TempCompass, and VidComposition, to demonstrate generalizability.

Key Insights and Ideas

  • The "Common Sense" Shortcut: The most striking insight is that current VLMMs often exploit prior knowledge of typical event sequences, rather than truly understanding the temporal order presented in the video. This is a significant limitation for tasks requiring nuanced temporal reasoning.
  • VECTOR's Design Philosophy: Creating a benchmark that forces models to rely on visual temporal cues by using synthetic, non-obvious event sequences is a clever way to isolate and measure true temporal understanding.
  • MECOT's Dual Approach: The combination of detailed, event-level instruction fine-tuning and structured CoT reasoning at inference is a powerful strategy. It addresses both the learning of temporal relationships and the explicit articulation of that reasoning.
  • Temporal Understanding is Hard: Even proprietary, large-scale models struggle significantly with temporal ordering, especially as the number of events increases, highlighting the difficulty of this task.

Key Results

  • VECTOR Performance: Various state-of-the-art VLMMs, including proprietary models like GPT-40 and Gemini 1.5 Pro, struggle on the VECTOR benchmark, showing significant performance drops compared to their performance on existing benchmarks.
  • MECOT's Superiority: MECOT consistently outperforms other open-source VLMMs on VECTOR across all tasks and difficulty levels. It also shows notable improvements on existing video benchmarks, demonstrating its effectiveness.
  • Proprietary Model Limitations: Even top-tier models like GPT-40 and Gemini 1.5 Pro show substantial performance declines on longer sequences (L2) in VECTOR, dropping from ~75-83% EM at L1 to ~50-60% at L2, and approach chance-level accuracy on longer sequences.
  • Impact of MECOT Components: Instruction fine-tuning (IFT) significantly boosts performance, while combining it with CoT further enhances it, indicating complementary benefits.

Limitations and Future Work

  • Human Performance Baseline: While proprietary models approach human performance on simpler tasks (L1), they still lag significantly on more complex temporal reasoning (L2), indicating a gap that MECOT aims to bridge.
  • Generalizability: While MECOT shows improvements on existing benchmarks, the paper focuses on its effectiveness on VECTOR. Further exploration of its performance across a wider range of complex temporal reasoning tasks could be beneficial.
  • Computational Cost: The paper mentions using 4x NVIDIA H100 GPUs for fine-tuning, suggesting that MECOT, while effective, might have significant computational requirements.
  • Future Work: The authors suggest that MECOT's success highlights the importance of explicit temporal reasoning. Future work could explore more sophisticated CoT strategies or other methods to further enhance VLMMs' ability to understand complex temporal dynamics in videos.

Summary updated

Abstract

Video Large Multimodal Models (VLMMs) have shown impressive performance in video understanding, yet their ability to accurately capture the temporal order of multiple events remains underexplored. We interestingly observe that, even when video frames are scrambled, models perform very well on the existing benchmarks by comprehensive experiments. This implies that VLMMs may not necessarily rely on accurate sequential processing of visual events, but instead depend on prior knowledge of typical scenarios to answer the question. To benchmark temporal understanding capabilities in VLMMs, we propose VECTOR, designed to explicitly assess a model’s ability to identify the temporal order of events. On this benchmark, we observe that various VLMMs often fail to understand the orders of events. To address this, we propose MECOT (Multi-Event instruction fine-tuning with Chain-of-Thought), which (1) trains models on detailed, event-by-event video descriptions and (2) using chain-ofthought prompts at inference to enhance temporal awareness. MECOT outperforms prior arts on VECTOR as well as improving performance on existing video benchmarks, implying effectiveness of temporal understanding. We release our code, model and datasets
BibTeX
@inproceedings{ahn2026what,
  author = {Daechul Ahn and Yura Choi and Hyeonbeom Choi and Seongwon Cho and San Kim and Jonghyun Choi},
  title = {{What Happens When: Learning Temporal Orders of Events in Videos}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2026},
  doi = {10.1109/wacv61042.2026.00273},
}
Figure