LongViTU: Instruction Tuning for Long-Form Video Understanding

Rujie Wu, Xiaojian Ma, Hai Ci, Yue Fan, Yuxuan Wang, Haozhe Zhao, Qing Li, Yizhou WangSubmission to ICLR 2025 · Withdrawn

TL;DR

LongViTU is a large-scale instruction-tuning dataset and benchmark designed to push Multimodal Language Models (MLMs) toward true long-form video comprehension. By organizing ~900 hours of video into a multi-scale hierarchical tree and applying an LLM-driven self-revision pipeline, the authors generate ~121k open-ended QA pairs with an average contextual reasoning length ("certificate length") of 4.6 minutes. Fine-tuning open-source models on LongViTU significantly narrows the performance gap with commercial systems like Gemini-1.5-Pro and yields substantial gains on external long-video benchmarks like EgoSchema (+30.7%).

What Problem Does This Paper Address?

While vision-language models excel at describing images and short video clips (<60 seconds), long-form video understanding remains a critical bottleneck. Existing automated video question-answering (VQA) datasets suffer from several shortcomings:

  1. Short temporal scope: Questions rarely require reasoning over long horizons (most average "certificate lengths" are under 60–100 seconds).
  2. Missing temporal ground truth: QA pairs lack explicit start-and-end event timestamps, making it difficult to pinpoint where models fail.
  3. Coarse scope and artificial tasks: Datasets often rely on multiple-choice formats, focus purely on static spatial questions (e.g., identifying objects in single frames), or contain verbose, hallucinated answers.
  4. Context explosion: Directly feeding long video frame sequences into LLMs quickly exceeds token limits and introduces excessive visual redundancy.

Methods

To overcome context limits during dataset generation, LongViTU introduces a three-stage automated synthesis pipeline:

[Frames (1 fps)] ──> InternLM-XComposer2 ──> Frame Captions 
                                                   │
[Human Annotations (Ego4D)] ───────────────────────┼──> GPT-4 ──> Event Descriptions
                                                                       │
                                                         GPT-4 ──> Segment Summaries
                                                                       │
                                                       [Hierarchical Video Tree]
                                                                       │
                                                       Sliding Window (5 segments)
                                                                       │
                                                        GPT-4 QA Generation (Q: first 3, A: last 2)
                                                                       │
                                                        GPT-4 Self-Revision ──> Final QA
  1. Hierarchical Video Tree Construction:
  2. Frame Level: Captions are sampled at 1 fps via InternLM-XComposer2 to capture fine-grained spatial and object details.
  3. Event Level: GPT-4 integrates frame captions with human-annotated event timestamps from Ego4D, eliminating redundancy into coherent event descriptions.
  4. Segment Level: GPT-4 groups consecutive events into macro-level segments.
  5. Sliding-Window QA Generation: A sliding window spans 5 segments. To simulate realistic temporal retrospection, GPT-4 poses questions based on events in the first three segments while deriving answers strictly from the subsequent two segments.
  6. Self-Revision Mechanism: GPT-4 evaluates generated QA pairs against the video context to eliminate hallucinations, strip out giveaway timestamps/indices, and simplify answers into concise, direct ground truths.
  7. Multi-Level Scoring Criteria: For evaluation, GPT-4 grades open-ended model answers against ground-truth references on a 6-tier rubric (0 to 100).

Datasets

  • LongViTU (Introduced): Built on the full Ego4D video suite (~900 hours across 1,833 egocentric videos averaging 29.3 minutes). Contains 121k QA pairs (101k train, 14k val, 6k test benchmark) split across 3 primary categories:
  • Spatiotemporal Understanding (55%): Object, Attribute, Location, Action.
  • Episodic Reasoning (24.4%): Transition, Interaction, Causality, Motivation.
  • Commonsense Inference (20.6%): Planning, Risk, Function, Affordance.
  • Evaluation Benchmarks: Evaluated internally on the LongViTU test set and externally on EgoSchema (In-Distribution), and VideoMME, WorldQA, and OpenEQA (Out-of-Distribution).

Key Insights and Ideas

  • Hierarchical tree structure beats flat context: Abstracting video from frames to events to segments allows LLMs to retain localized spatial fidelity while understanding long-range event causality, bypassing context-window degradation.
  • Fine-grained spatial reasoning is harder than temporal logic: Models scored markedly lower on Spatiotemporal Understanding (Video-LLaVA: 44.9) than on Episodic Reasoning (58.0) and Commonsense Inference (59.8). The sequential nature of LLMs naturally captures narrative logic, but dense spatial localization remains a bottleneck.
  • The Blind-QA reality check: Pure text-based GPT-4 without video input achieved an overall score of 38.2. This confirms that text-mediated dataset generation can inherit language priors, emphasizing the need for rigorous multi-modal grounding.

Key Results

  • Closing the Gap with Commercial Models: On the LongViTU benchmark, supervised fine-tuning (SFT) boosted Video-LLaVA from 45.9 to 50.7, closely approaching the proprietary Gemini-1.5-Pro (52.3 at 1 fps). LLaMA-VID improved from 38.2 to 44.5 (+16.5%).
  • Cross-Benchmark Generalization:
  • EgoSchema (ID): SFT Video-LLaVA jumped from 36.8 to 48.1 (+30.7% relative improvement).
  • WorldQA (OOD): Increased from 30.2 to 34.1 (+12.9%).
  • VideoMME (OOD): Overall improved by +0.6%, with the strongest gains on Long videos (+5.1%) and Medium videos (+6.6%).

Limitations and Future Work

  • Text Intermediary Bias: Because dataset generation uses textual frame descriptions as an intermediary, linguistic shortcuts can inadvertently be introduced into the QA pairs.
  • Context Utility: Automated generation can occasionally produce QA pairs that are factually accurate but lack deep conversational utility. The authors suggest future iterations will add manual human verification.
  • Future Directions: Incorporating streaming video architectures and exploring specialized video memory mechanisms for long-horizon egocentric AI assistants.

Summary updated

Abstract

This paper presents LongViTU, a large-scale (~121k QA pairs, ~900h videos), automatically generated dataset for long-form video understanding. Our key idea is inspired by the success of Large Language Models (LLMs) and Multimodal Language Models (MLMs) that are fueled by machine-generated instruction-following data (*e.g.*, InstructGPT, LLaVA). We developed a *systematic* approach to produce massive question-answeringing pairs tailored to virtually unbounded long videos by organizing them into a ***hierarchical tree***, incorporating ***self-revision*** mechanisms to guarantee high quality. We curate LongViTU for each QA pair: 1) involves a long context (average *certificate length* of 4.6 minutes); 2) requires rich knowledge and condensed reasoning (commonsense, causality, planning, *etc.*); 3) explicit labels the timestamps of relevant events throughout the entire video. Furthermore, LongViTU provides a benchmark to facilitate future research in instruction-following for long-form videos. Our experiments first reveal the performance gap between open-source video MLMs and their commercial counterparts (*e.g.*, Gemini-1.5-Pro) on this benchmark. Supervised Fine-Tuning (SFT) on open-source models led to Video-LLaVA achieving the best performance, with a GPT-4 score of $50.7$, closely following $52.3$ by the leading closed-source model Gemini-1.5-Pro, underscoring the substantial challenge posed by our benchmark. Further SFT on LongViTU with Video-LLaVA resulted in improvements of $30.7$% on the In-Distribution (ID) benchmark EgoSchema; $12.9$% and $0.6$% on the Out-of-Distribution (OOD) benchmarks WorldQA and VideoMME, respectively. These outcomes demonstrate the effectiveness and robust OOD generalizability of our proposed instruction-tuning scheme for long-form video understanding. The dataset, SFT models, and code are publicly available on the anonymous page [LongViTU](https://longvitu.github.io).
BibTeX
@misc{wu2025longvitu,
  author = {Rujie Wu and Xiaojian Ma and Hai Ci and Yue Fan and Yuxuan Wang and Haozhe Zhao and Qing Li and Yizhou Wang},
  title = {{LongViTU: Instruction Tuning for Long-Form Video Understanding}},
  year = {2025},
  eprint = {2501.05037},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.05037},
}
Figure