TL;DR
The paper introduces HLV-1K, the largest benchmark dataset designed specifically for hour-long video understanding. Comprising 1,009 videos and nearly 15,000 time-specific question-answering pairs, it challenges multimodal models across various reasoning levels. The evaluation reveals a surprising twist: specialized open-source models significantly outperform massive commercial models like GPT-4o and Gemini 1.5 Pro on long-context video tasks.
What Problem Does This Paper Address?
While Multimodal Large Language Models (MM-LLMs) excel at analyzing images and short video clips, understanding hour-long videos remains a massive hurdle. Long videos contain tens of thousands of frames, introducing noise, redundancy, and severe memory/computation constraints.
Current video benchmarks mostly focus on second- or minute-level clips. The few datasets that do feature longer videos often lack precise temporal annotations or rely solely on multiple-choice questions. Without a large-scale, granular benchmark, it is difficult to rigorously evaluate how well a model maintains long-term dependencies or navigates complex temporal dynamics.
Methods
To build a high-quality dataset without manual brute force, the authors developed a structured, four-step semi-automated annotation pipeline:
- Frame Extraction: To eliminate massive redundancy, raw videos are compressed to 1 frame per second (fps). Keyframes are then extracted using I-frame detection methods, resulting in an manageable average of 810 frames per video.
- Frame Labeling: GPT-4o generates detailed descriptions for each frame. To ensure accuracy, the model is fed conditional information: the frame's timestamp and object bounding boxes generated by a YOLOv8 object detector.
- Event Labeling: A sliding-window approach (window size of 100 frames, ~10 minutes) groups coherent frames into distinct events, which GPT-4o then summarizes.
- QA Generation: Using the frame and event descriptions, the system generates time-aware Question Answering (QA) and Multiple-Choice QA (MCQA) pairs across four reasoning granularities: frame-level, within-event-level, cross-event-level, and long-term-level.
Finally, a rigorous manual filtering and revision stage ensures that all timestamps are accurate and that questions contain no formatting errors or missing answers.
Datasets
The authors introduce HLV-1K, which contains:
- 1,009 long videos sourced from YouTube (based on the HD-VILA dataset), each exceeding 30 minutes, with an average duration of 55 minutes.
- 14,847 high-quality QA and MCQA pairs.
- Diverse content spanning 9 categories, heavily featuring Film (33.2%), Animation (16.1%), and Travel (12.5%).
Unlike previous benchmarks like Video-MME or LongVideoBench, every single query in HLV-1K is explicitly linked to precise timestamps, forcing models to demonstrate true temporal perception.
Key Insights and Ideas
- Open-Source Specialization Beats Commercial Scale: Massive proprietary models like GPT-4o and Gemini 1.5 Pro are often seen as the gold standard, but they fell significantly behind specialized open-source video models (like LLaVA-Video) on this benchmark.
- The Frame Budget Bottleneck: Performance drops drastically when a model's frame capacity is restricted. For instance, Claude 3.5 Sonnet—which was limited to processing only 20 frames—struggled immensely, highlighting that temporal density is non-negotiable for long video comprehension.
- Universal Blind Spots: While some models excel at identifying objects or describing scenes, all models struggle with tasks requiring deep temporal or mathematical reasoning, such as tracking time-order, understanding causal relationships, counting, and identifying camera directions.
Key Results
The authors evaluated several state-of-the-art models by uniformly sampling 120 frames per video (except for Claude 3.5 Sonnet, which was capped at 20 frames):
- LLaVA-Video (72B) achieved the top overall score of 78.93%, showing superior capability in handling complex, long-term dependencies.
- LLaVA-OneVision (72B) followed in second place with 74.01%.
- Gemini 1.5 Pro scored 62.41%, noticeably underperforming compared to the top open-source models.
- GPT-4o scored 55.48%, struggling particularly on long-term reasoning tasks.
- Claude 3.5 Sonnet scored just 27.24%, severely handicapped by its 20-frame input limit.
As expected, long-term-level tasks proved to be the most challenging across the board, while frame-level spatial reasoning yielded the highest accuracies.
Limitations and Future Work
The authors note a practical evaluation challenge: to prevent information leakage across questions, each QA pair should ideally be evaluated independently. However, because this is computationally expensive for hour-long videos, they resorted to batching questions together.
Additionally, the benchmark highlights a clear architectural limitation in current MM-LLMs: their fundamental weakness in complex spatial-temporal reasoning (like counting and tracking sequential order over time). The authors hope HLV-1K will serve as a catalyst for developing next-generation models capable of truly understanding long live streams, meeting recordings, and full-length movies.
Summary updated