HLV-1K: A Large-scale Hour-Long Video Benchmark for Time-Specific Long Video Understanding

Heqing Zou, Tianze Luo, Guiyang Xie, Victor Xiao Jie Zhang, Fengmao Lv, Guangcong Wang, Junyang Chen, Zhuochen Wang, Hansheng Zhang, Huaijian Zhang

Abstract

While Multimodal Large Language Models (MM-LLMs) have demonstrated remarkable capabilities in image and short-clip analysis, understanding hour-long video remains a significant challenge. This paper introduces HLV-1K, a large-scale benchmark designed to evaluate long-form video comprehension. Comprising over 1,000 videos and approximately 15,000 time-aware question-answer pairs, HLV-1K reveals that specialized open-source models currently outperform general-purpose commercial models in long-term temporal reasoning.

Introduction

Most existing video benchmarks focus on short clips ranging from seconds to a few minutes. However, real-world applications—such as analyzing meeting recordings, feature-length films, or live streams—require models to process content exceeding an hour. This introduces substantial hurdles: maintaining long-term memory, managing complex temporal dynamics, and processing tens of thousands of frames within compute constraints. HLV-1K addresses the lack of high-quality, large-scale benchmarks for hour-long videos, providing a reliable metric for measuring how well models follow extended narratives.

Methods

To manage the scale of hour-long data, the authors developed a sophisticated, hierarchical annotation pipeline. The process involves several key stages to transform raw video into structured, time-specific queries.

  1. Smart Frame Extraction: Videos are compressed to 1 frame-per-second, and keyframes (I-frames) are extracted to reduce redundancy while preserving visual information, averaging 810 frames per video.
  2. AI-Assisted Labeling: Object detection via YOLOv8 provides bounding boxes and timestamps, which are processed by GPT-4o to generate detailed frame descriptions.
  3. Event Grouping: A sliding window approach aggregates frame descriptions into broader "event" summaries.
  4. Multi-Level QA Generation: Question-answering (QA) and multiple-choice (MCQA) pairs are generated across four difficulty tiers: frame-level, within-event, cross-event, and long-term reasoning.
  5. Human Filtering: All AI-generated content undergoes manual review to ensure accuracy and precise timestamp alignment.
The hierarchical data annotation pipeline for HLV-1K, spanning collection, multi-stage annotation, and final revision.

Dataset Characteristics

HLV-1K consists of 1,009 videos with an average duration of 55 minutes, totaling 14,847 high-quality QA pairs. It significantly outscales previous long-video datasets like HourVideo (500 videos) and LVBench (103 videos). A defining feature of HLV-1K is that every query is time-specific, explicitly testing temporal reasoning rather than general global summaries.

Dataset statistics for HLV-1K, including category distribution (a), video duration (b), and query duration (c).

Key Results and Insights

Evaluations of state-of-the-art models on HLV-1K yielded several critical findings:

  • Open-Source Superiority: Specialized models like LLaVA-Video (72B) achieved the highest overall accuracy (78.93%), outperforming commercial giants like Gemini 1.5 Pro (62.41%) and GPT-4o (55.48%).
  • The Frame Bottleneck: Performance is heavily dependent on input density. Claude 3.5 Sonnet, capped at 20 frames, scored only 27.24%, proving that sparse sampling is insufficient for hour-long content.
  • The "Temporal Wall": While models perform well at frame-level identification, they struggle with tasks requiring deep temporal/spatial reasoning, such as causality, counting, and tracking camera direction across events.
  • Reasoning Hierarchy: The benchmark confirms that cross-event and long-term reasoning are fundamentally more difficult for current architectures than isolated event comprehension.

Conclusion

HLV-1K exposes a clear gap in current MM-LLM capabilities regarding long-form temporal and spatial reasoning. Future work must focus on developing architectures capable of tracking causality and time over thousands of frames. While the dataset relies on AI for initial generation, its human-filtered, multi-level structure provides a rigorous roadmap for the next generation of video-centric models.

Summary updated

BibTeX
@inproceedings{zou2025hlv,
  author = {Heqing Zou and Tianze Luo and Guiyang Xie and Victor Xiao Jie Zhang and Fengmao Lv and Guangcong Wang and Junyang Chen and Zhuochen Wang and Hansheng Zhang and Huaijian Zhang},
  title = {{HLV-1K: A Large-scale Hour-Long Video Benchmark for Time-Specific Long Video Understanding}},
  booktitle = {ICME},
  year = {2025},
  doi = {10.1109/icme59968.2025.11208965},
}
Figure