HLV-1K: A Large-scale Hour-Long Video Benchmark for Time-Specific Long Video Understanding

Heqing Zou, Tianze Luo, Guiyang Xie, Victor Xiao Jie Zhang, Fengmao Lv, Guangcong Wang, Junyang Chen, Zhuochen Wang, Hansheng Zhang, Huaijian Zhang

TL;DR

The paper introduces HLV-1K, the largest benchmark dataset designed specifically for hour-long video understanding. Comprising 1,009 videos and nearly 15,000 time-specific question-answering pairs, it challenges multimodal models across various reasoning levels. The evaluation reveals a surprising twist: specialized open-source models significantly outperform massive commercial models like GPT-4o and Gemini 1.5 Pro on long-context video tasks.

What Problem Does This Paper Address?

While Multimodal Large Language Models (MM-LLMs) excel at analyzing images and short video clips, understanding hour-long videos remains a massive hurdle. Long videos contain tens of thousands of frames, introducing noise, redundancy, and severe memory/computation constraints.

Current video benchmarks mostly focus on second- or minute-level clips. The few datasets that do feature longer videos often lack precise temporal annotations or rely solely on multiple-choice questions. Without a large-scale, granular benchmark, it is difficult to rigorously evaluate how well a model maintains long-term dependencies or navigates complex temporal dynamics.

Methods

To build a high-quality dataset without manual brute force, the authors developed a structured, four-step semi-automated annotation pipeline:

  1. Frame Extraction: To eliminate massive redundancy, raw videos are compressed to 1 frame per second (fps). Keyframes are then extracted using I-frame detection methods, resulting in an manageable average of 810 frames per video.
  2. Frame Labeling: GPT-4o generates detailed descriptions for each frame. To ensure accuracy, the model is fed conditional information: the frame's timestamp and object bounding boxes generated by a YOLOv8 object detector.
  3. Event Labeling: A sliding-window approach (window size of 100 frames, ~10 minutes) groups coherent frames into distinct events, which GPT-4o then summarizes.
  4. QA Generation: Using the frame and event descriptions, the system generates time-aware Question Answering (QA) and Multiple-Choice QA (MCQA) pairs across four reasoning granularities: frame-level, within-event-level, cross-event-level, and long-term-level.

Finally, a rigorous manual filtering and revision stage ensures that all timestamps are accurate and that questions contain no formatting errors or missing answers.

Fig. 2: Construction of HLV-1K: (a) HLV-1K construction pipeline with data collection, data labeling and data filtering and revision, (b) Case of QA sample in HLV-1K and (b) Case of MCQA sample in HLV-1K.

Datasets

The authors introduce HLV-1K, which contains:

  • 1,009 long videos sourced from YouTube (based on the HD-VILA dataset), each exceeding 30 minutes, with an average duration of 55 minutes.
  • 14,847 high-quality QA and MCQA pairs.
  • Diverse content spanning 9 categories, heavily featuring Film (33.2%), Animation (16.1%), and Travel (12.5%).

Unlike previous benchmarks like Video-MME or LongVideoBench, every single query in HLV-1K is explicitly linked to precise timestamps, forcing models to demonstrate true temporal perception.

Fig. 1: HLV-1K statistics: (a) Video category distribution, (b) Video duration distribution, and (c) Duration distribution of time-specific queries.

Key Insights and Ideas

  • Open-Source Specialization Beats Commercial Scale: Massive proprietary models like GPT-4o and Gemini 1.5 Pro are often seen as the gold standard, but they fell significantly behind specialized open-source video models (like LLaVA-Video) on this benchmark.
  • The Frame Budget Bottleneck: Performance drops drastically when a model's frame capacity is restricted. For instance, Claude 3.5 Sonnet—which was limited to processing only 20 frames—struggled immensely, highlighting that temporal density is non-negotiable for long video comprehension.
  • Universal Blind Spots: While some models excel at identifying objects or describing scenes, all models struggle with tasks requiring deep temporal or mathematical reasoning, such as tracking time-order, understanding causal relationships, counting, and identifying camera directions.

Key Results

The authors evaluated several state-of-the-art models by uniformly sampling 120 frames per video (except for Claude 3.5 Sonnet, which was capped at 20 frames):

  • LLaVA-Video (72B) achieved the top overall score of 78.93%, showing superior capability in handling complex, long-term dependencies.
  • LLaVA-OneVision (72B) followed in second place with 74.01%.
  • Gemini 1.5 Pro scored 62.41%, noticeably underperforming compared to the top open-source models.
  • GPT-4o scored 55.48%, struggling particularly on long-term reasoning tasks.
  • Claude 3.5 Sonnet scored just 27.24%, severely handicapped by its 20-frame input limit.

As expected, long-term-level tasks proved to be the most challenging across the board, while frame-level spatial reasoning yielded the highest accuracies.

Limitations and Future Work

The authors note a practical evaluation challenge: to prevent information leakage across questions, each QA pair should ideally be evaluated independently. However, because this is computationally expensive for hour-long videos, they resorted to batching questions together.

Additionally, the benchmark highlights a clear architectural limitation in current MM-LLMs: their fundamental weakness in complex spatial-temporal reasoning (like counting and tracking sequential order over time). The authors hope HLV-1K will serve as a catalyst for developing next-generation models capable of truly understanding long live streams, meeting recordings, and full-length movies.

Summary updated

Abstract

Multimodal large language models have become a popular topic in deep visual understanding due to many promising real-world applications. However, hour-long video understanding, spanning over one hour and containing tens of thousands of visual frames, remains under-explored because of 1) challenging long-term video analyses, 2) inefficient large-model approaches, and 3) lack of large-scale benchmark datasets. Among them, in this paper, we focus on building a large-scale hour-long long video benchmark, HLV-1K, designed to evaluate long video understanding models. HLV-1K comprises 1009 hour-long videos with 14,847 high-quality question answering (QA) and multi-choice question asnwering (MCQA) pairs with time-aware query and diverse annotations, covering frame-level, within-event-level, cross-event-level, and long-term reasoning tasks. We evaluate our benchmark using existing state-of-the-art methods and demonstrate its value for testing deep long video understanding capabilities at different levels and for various tasks. This includes promoting future long video understanding tasks at a granular level, such as deep understanding of long live videos, meeting recordings, and movies.
BibTeX
@inproceedings{zou2025hlv,
  author = {Heqing Zou and Tianze Luo and Guiyang Xie and Victor Xiao Jie Zhang and Fengmao Lv and Guangcong Wang and Junyang Chen and Zhuochen Wang and Hansheng Zhang and Huaijian Zhang},
  title = {{HLV-1K: A Large-scale Hour-Long Video Benchmark for Time-Specific Long Video Understanding}},
  booktitle = {ICME},
  year = {2025},
  doi = {10.1109/icme59968.2025.11208965},
}
Figure