Abstract
The recent success of Large Language Models (LLMs) has prompted the extension to the multimodal domain, developing image-text Multimodal LLMs (MLLMs) and then video-text models. In this work, we investigate the challenge of contextual and temporal comprehension in video-language models by exploring the task of temporal localization in videos. To address this problem, prior works have developed complex task-specific architectures, novel modules to embed time into MLLMs, or leveraged additional input signals such as video transcripts to best encode contextual and temporal information. We find that most of these efforts are surpassed by a much simpler design. We introduce Chrono, a universal sequence blueprint that can be applied to any image-text pretrained MLLM. In extensive experiments spanning different MLLM architectures and sizes, finetuning and zero-shot settings, we demonstrate new state-of-the-art results in moment retrieval on the widely used benchmarks Charades-STA, QVHighlights, and ActivityNet Captions, as well as in grounded video question answering on NExT-GQA.
Introduction
While Multimodal Large Language Models (MLLMs) have achieved remarkable success in static image understanding, extending this capability to the temporal dimension of video remains a significant challenge. Specifically, tasks like Temporal Localization—finding the exact start and end times of an event within a video—require a native sense of time that standard image-text models lack.
Historically, researchers have addressed this by engineering specialized temporal perception modules, time-aware frame encoders, or relying on external signals like video transcripts. However, these solutions often introduce architectural complexity and require massive video-specific pretraining. Chrono challenges this paradigm by demonstrating that the bottleneck is not the model's architecture, but rather the data format used to represent time.
The Chrono Blueprint
The core innovation of this work is the Chrono blueprint, a model-agnostic method for formatting input sequences that enables any image-text MLLM to reason about time without architectural changes.
Key Design Principles
- Interleaved Timestamps: Instead of treating time as a separate metadata field, Chrono interleaves sampled video frames with their corresponding timestamps directly in the input sequence (e.g.,
[Frame 1], "0 seconds", [Frame 2], "30 seconds"). This allows the model's attention mechanism to explicitly link visual features to temporal coordinates. - Integer Precision: The authors discovered that rounding timestamps to the nearest whole second significantly outperforms decimal representations. This is attributed to how LLM tokenizers handle numbers; integers are typically represented as single, clean tokens, whereas decimals can be split into multiple confusing sub-tokens.
- Sequence-to-Sequence Output: The model is trained to predict temporal boundaries as simple text strings (e.g.,
[[12, 18]]), leveraging the model's existing language generation capabilities for localization.
Experimental Results
Chrono was evaluated across four major benchmarks: Charades-STA, QVHighlights, ActivityNet Captions, and NExT-GQA.
- State-of-the-Art Performance: Chrono-BLIP achieved a new SOTA on QVHighlights with 74.77 [email protected], outperforming much larger and more complex models like InternVideo2.
- Zero-Shot Capabilities: When applied to closed-source models like GPT-4o, the Chrono blueprint enabled a 36.5 mIoU on NExT-GQA, nearly doubling the performance of previous specialized models like SeViLA (21.7).
- Generalization: Chrono demonstrated superior cross-dataset transfer, maintaining high performance even when tested on datasets different from its training source.
Discussion and Limitations
The success of Chrono suggests that "simplicity is sovereignty." By using the native language space of the MLLM to represent time, the blueprint avoids the need for task-specific engineering.
However, limitations remain. The current approach relies on uniform frame sampling, which may miss very brief events in long videos (sampling aliasing). Additionally, the model is still bound by the resolution limits of its underlying vision encoder. Future work will explore adaptive sampling techniques and scaling the blueprint to handle even longer video contexts and more complex multi-step reasoning tasks.
Summary updated