TEMPURA: Temporal Event Masked Prediction and Understanding for Reasoning in Action

Jen-Hao Cheng, Vivian Wang, Huayu Wang, Huapeng Zhou, Yi-Hao Peng, Hou-I Liu, Hsiang-Wei Huang, Kuang-Ming Chen, Cheng-Yen Yang, Wenhao Chai, Yiling Chen, Vibhav Vineet, Qin Cai, Jenq-Neng Hwang

TL;DR

TEMPURA introduces a novel two-stage training framework that teaches Video Large Multi-modal Models (LMMs) to understand the flow of time and causality in videos by having them "fill in the blanks" of missing events. Paired with a massive new dataset of 500,000 densely captioned videos, TEMPURA significantly boosts a model's ability to pinpoint specific moments and understand the logical sequence of actions, outperforming larger baseline models without needing task-specific fine-tuning.

What Problem Does This Paper Address?

While modern Vision-Language Models are getting incredibly good at answering questions about static images or short video clips, they easily get lost in long videos. Specifically, they struggle with temporal reasoning—understanding the order of events, pinpointing exact start and end times (temporal grounding), and grasping causal relationships (e.g., realizing that a chef must batter the shrimp before frying it).

Current approaches try to solve this by either compressing video frames to save memory (which destroys fine-grained temporal details) or treating the video as one continuous, unsegmented stream (which obscures the boundaries between distinct events). The authors of this paper tackle the challenge of teaching an LMM to naturally segment long videos into distinct, logical steps and understand the causal links connecting them.

Methods

The authors propose TEMPURA, a two-stage training pipeline built on top of the Qwen2.5-VL (3B) model. The framework focuses on bridging the gap between raw video perception and logical event sequencing.

Overview of the TEMPURA framework, illustrating masked reasoning, video segmentation, and the VER dataset.

Stage 1: Masked Event Prediction Reasoning

Inspired by "Fill-in-the-Middle" techniques used in text-based LLMs, TEMPURA applies this to video. The model is given a video sequence with a chunk of time masked out. It must use the surrounding context to predict what happened during the missing time and generate a step-by-step logical explanation for why it thinks that event occurred.

Detail of the Masked Event Reasoning task, showing how the model predicts missing temporal segments in a video.

Stage 2: Video Event Segmentation and Temporal Dense Captioning

Next, the model is trained to partition untrimmed videos into a series of non-overlapping events. It learns to output precise start and end timestamps for each segment alongside a detailed text description.

A Clever Engineering Trick: To help the model keep track of time in long videos, the authors found that standard mathematical positional encodings often failed. Their solution? They literally overlaid visual timestamps (text) onto the upper left corner of the sampled video frames, allowing the model's strong OCR (Optical Character Recognition) capabilities to naturally track the video's progression.

Datasets

  • VER (Video Event Reasoning) Dataset: Because existing datasets lack dense, timestamp-aligned causal reasoning data, the authors built VER. It contains 500,000 untrimmed videos (18,000 hours) across 10 categories. They used GPT-4o to filter out static scenes, segment the videos, generate dense captions, and create the synthetic "masked event" reasoning pairs used in Stage 1.
  • Evaluation Benchmarks: The model was tested on Charades-STA (for video temporal grounding) and QVHighlights (for highlight detection).

Key Insights and Ideas

  • Order of operations matters: Ablation studies reveal that teaching the model causal reasoning (Stage 1) before segmentation and captioning (Stage 2) is critical. Without the logical foundation first, the model fails to learn fine-grained temporal grounding.
  • Visual timestamps beat complex math: Simply stamping the time onto the video frames proved to be the most robust way to encode time, outperforming complex absolute temporal encoding schemes like M-RoPE, especially as videos get longer.
  • Curating for causality: Training on random video segments is ineffective. Explicitly filtering the training data to remove static scenes and ensuring sequential events have a causal relationship was critical for learning bidirectional temporal thinking.

Key Results

Despite using a relatively small 3-Billion parameter base model, TEMPURA achieved impressive zero-shot results:

  • Video Temporal Grounding (Charades-STA): Achieved a mIoU of 39.2, outperforming the Qwen2.5-VL baseline by 6.3 points.
  • Highlight Detection (QVHighlights): Reached a HIT@1 score of 51.7 (a 6.9 point jump over the baseline) and a mAP of 48.3.
  • Qualitative leaps: Compared to baselines, TEMPURA generates vastly more granular event descriptions. On the Youcook2 dataset, TEMPURA identified 27.5 distinct, logically sequenced events per video, compared to the baseline's 15.5.

Limitations and Future Work

  • Reliance on Synthetic Data: The VER dataset relies on GPT-4o for generating ground truth captions and reasoning. Any hallucinations or temporal misunderstandings in the teacher model will be inherited by TEMPURA.
  • Visual Timestamp Hack: While effective, overlaying text timestamps is a workaround. Future work should focus on native temporal encoding architectures that don't require altering the raw visual input.

Summary updated

BibTeX
@inproceedings{cheng2026tempura,
  author = {Jen-Hao Cheng and Vivian Wang and Huayu Wang and Huapeng Zhou and Yi-Hao Peng and Hou-I Liu and Hsiang-Wei Huang and Kuang-Ming Chen and Cheng-Yen Yang and Wenhao Chai and Yiling Chen and Vibhav Vineet and Qin Cai and Jenq-Neng Hwang},
  title = {{TEMPURA: Temporal Event Masked Prediction and Understanding for Reasoning in Action}},
  booktitle = {COLM},
  year = {2026},
  url = {https://openreview.net/forum?id=8GwVxWwPQr},
}
Figure