Chrono: A Simple Blueprint for Representing Time in MLLMs

Batra Anil, Hector G. Rodriguez, Anil Batra, Anna Rohrbach, Marcus Rohrbach

Abstract

The recent success of Large Language Models (LLMs) has prompted the extension to the multimodal domain, developing image-text Multimodal LLMs (MLLMs) and then video-text models. In this work, we investigate the challenge of contextual and temporal comprehension in video-language models by exploring the task of temporal localization in videos. To address this problem, prior works have developed complex task-specific architectures, novel modules to embed time into MLLMs, or leveraged additional input signals such as video transcripts to best encode contextual and temporal information. We find that most of these efforts are surpassed by a much simpler design. We introduce Chrono, a universal sequence blueprint that can be applied to any image-text pretrained MLLM. In extensive experiments spanning different MLLM architectures and sizes, finetuning and zero-shot settings, we demonstrate new state-of-the-art results in moment retrieval on the widely used benchmarks Charades-STA, QVHighlights, and ActivityNet Captions, as well as in grounded video question answering on NExT-GQA.

Introduction

While Multimodal Large Language Models (MLLMs) have achieved remarkable success in static image understanding, extending this capability to the temporal dimension of video remains a significant challenge. Specifically, tasks like Temporal Localization—finding the exact start and end times of an event within a video—require a native sense of time that standard image-text models lack.

Chrono simplifies temporal modeling compared to prior complex architectures.

Historically, researchers have addressed this by engineering specialized temporal perception modules, time-aware frame encoders, or relying on external signals like video transcripts. However, these solutions often introduce architectural complexity and require massive video-specific pretraining. Chrono challenges this paradigm by demonstrating that the bottleneck is not the model's architecture, but rather the data format used to represent time.

The Chrono Blueprint

The core innovation of this work is the Chrono blueprint, a model-agnostic method for formatting input sequences that enables any image-text MLLM to reason about time without architectural changes.

The Chrono input sequence interleaves visual tokens with text-based integer timestamps.

Key Design Principles

  1. Interleaved Timestamps: Instead of treating time as a separate metadata field, Chrono interleaves sampled video frames with their corresponding timestamps directly in the input sequence (e.g., [Frame 1], "0 seconds", [Frame 2], "30 seconds"). This allows the model's attention mechanism to explicitly link visual features to temporal coordinates.
  2. Integer Precision: The authors discovered that rounding timestamps to the nearest whole second significantly outperforms decimal representations. This is attributed to how LLM tokenizers handle numbers; integers are typically represented as single, clean tokens, whereas decimals can be split into multiple confusing sub-tokens.
  3. Sequence-to-Sequence Output: The model is trained to predict temporal boundaries as simple text strings (e.g., [[12, 18]]), leveraging the model's existing language generation capabilities for localization.

Experimental Results

Chrono was evaluated across four major benchmarks: Charades-STA, QVHighlights, ActivityNet Captions, and NExT-GQA.

  • State-of-the-Art Performance: Chrono-BLIP achieved a new SOTA on QVHighlights with 74.77 [email protected], outperforming much larger and more complex models like InternVideo2.
  • Zero-Shot Capabilities: When applied to closed-source models like GPT-4o, the Chrono blueprint enabled a 36.5 mIoU on NExT-GQA, nearly doubling the performance of previous specialized models like SeViLA (21.7).
  • Generalization: Chrono demonstrated superior cross-dataset transfer, maintaining high performance even when tested on datasets different from its training source.

Discussion and Limitations

The success of Chrono suggests that "simplicity is sovereignty." By using the native language space of the MLLM to represent time, the blueprint avoids the need for task-specific engineering.

However, limitations remain. The current approach relies on uniform frame sampling, which may miss very brief events in long videos (sampling aliasing). Additionally, the model is still bound by the resolution limits of its underlying vision encoder. Future work will explore adaptive sampling techniques and scaling the blueprint to handle even longer video contexts and more complex multi-step reasoning tasks.

Summary updated

Abstract

The recent success of Large Language Models (LLMs) has prompted the extension to the multimodal domain, developing image-text Multimodal LLMs (MLLMs) and then videotext models. In this work, we investigate the challenge of contextual and temporal comprehension in video-language models by exploring the task of temporal localization in videos. To address this problem, prior works have developed complex task-specific architectures, novel modules to embed time into MLLMs, or leveraged additional input signals such as video transcripts to best encode contextual and temporal information. Interestingly, we find that most of these efforts are surpassed by a much simpler design. We introduce Chrono , a universal sequence blueprint that can be applied to an image-text pretrained MLLM. Through extensive ablations across different MLLM architectures, finetuning and zero-shot settings, and different datasets, we achieve a new SOTA in moment retrieval on the most widely used benchmarks Charades-STA, QVHighlights, ActivityNet Captions, and grounded video question answering on NeXT-GQA. Our code will be made publicly available.
BibTeX
@misc{anil2024chrono,
  author = {Batra Anil and Hector G. Rodriguez and Anil Batra and Anna Rohrbach and Marcus Rohrbach},
  title = {{Chrono: A Simple Blueprint for Representing Time in MLLMs}},
  year = {2024},
  eprint = {2406.18113},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.18113},
}
Figure