Temporal Reasoning Decay in Long Video Understanding

38 concepts29 papers1 proposal

While modern Multimodal Large Language Models (MLLMs) can describe short clips with startling accuracy, their ability to reason over time is fragile. As video duration extends from seconds to minutes or hours, a phenomenon known as temporal reasoning decay emerges. Models that appear intelligent in brief windows often revert to near-random guessing or language-based hallucinations when tasked with maintaining context over long sequences.

The Long Context Gap

The scale of this failure is best illustrated by the "long-context gap." In HourVideo, Keshigeyan Chandrasegaran et al. demonstrate that while human experts achieve 85% accuracy on hour-long video tasks, state-of-the-art models like Gemini 1.5 Pro struggle at roughly 37%. This isn't merely a lack of data; as Junjie Zhou et al. observe in MLVU, performance often hits a "cliff" after just ten minutes of footage, suggesting that current architectures cannot maintain coherent internal representations as visual context accumulates.

The Causal Distance Penalty

A primary driver of this decay is the temporal distance penalty: the increasing difficulty of linking two related events as the time between them grows. In Two Causally Related Needles in a Video Haystack, Boyang Li et al. show that while humans easily connect a cause to an effect regardless of intervening time, model accuracy collapses as the gap between these "needles" increases.

This suggests that models lack a persistent internal state. Instead of "understanding" a physical change, they rely on short-term visual patterns. Maitreya Patel et al. argue in CRIPP-VQA that physical reasoning decays because models fail to maintain a latent representation of an object’s state, such as its velocity or hidden properties, over sequential interactions.

The U-Shaped Performance Curve

Intuitively, one might expect reasoning to get harder linearly as a video gets longer. However, David Ma et al. identified a surprising U-shaped performance curve in ScaleLong. Models are generally effective at two extremes:

  1. Short-term actions: Recognizing a quick movement (e.g., "picking up a glass").
  2. Global summaries: Identifying the overall theme of a video (e.g., "this is a cooking tutorial").

The decay is most acute in the mid-range events—the intermediate narrative steps that connect local actions to the global plot. This indicates an architectural deficit in stitching together mid-range temporal coherence, where the model loses the logical thread between the "how" of a specific frame and the "why" of the entire sequence.

Intent Dilution and the Forgetting Phenomenon

Even when a model is given a specific anchor to look for, long-form video causes intent dilution. In SAM2-LOVE, Yong Liu et al. describe how the sequential propagation of multimodal tokens leads to the gradual loss of the original query's context.

This is compounded by a forgetting phenomenon related to input order. In IV-Bench, David Ma et al. found that if a visual query (like an image of a specific object) is placed at the start of a long video stream, the model often "forgets" its relevance by the time it reaches the end of the footage. The sheer volume of subsequent video tokens acts as noise, drowning out the initial prompt.

Why Current Mechanisms Fail

The evidence across these benchmarks suggests that simply increasing context windows is insufficient. Xi Ding et al. argue in Do Language Models Understand Time? that current tokenization and memory mechanisms are fundamentally ill-equipped for extended video. Because models often "fake" understanding through semantic pattern matching rather than true temporal logic, they struggle with:

  • Chronological Dependencies: Enforcing that Event A must happen before Event B (Harsh Goel et al.).
  • Incremental Reasoning: Updating beliefs as new information arrives in a stream (Yuxuan Wang et al.).
  • Information Density: Managing the high noise-to-signal ratio in hour-long documentaries (Chaoyou Fu et al.).

To overcome temporal reasoning decay, the field is shifting toward methods that explicitly align reasoning steps with the temporal flow, such as the process-based rewards proposed by Sicheng Tao et al. in MOSS-ChatV, aiming to ensure models arrive at the right answer for the right temporal reasons.

Key Foundational Work

Learning Temporal Video Procedure Segmentation from an Automatically Collected Large Dataset (2022) — contributed 2 concept elements to this direction.
CRIPP-VQA: Counterfactual Reasoning about Implicit Physical Properties via Video Question Answering (2022) — contributed 2 concept elements to this direction.
Env-QA: A Video Question Answering Benchmark for Comprehensive Understanding of Dynamic Environments (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_temporal_reasoning_decay_dda96b21,
  title = {temporal reasoning decay},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-reasoning-decay/rdr_dda96b21bd3e438b88e6583d295acacd},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_temporal_reasoning_decay_dda96b21_v1,
  title = {temporal reasoning decay},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-reasoning-decay/rdr_dda96b21bd3e438b88e6583d295acacd/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake