TL;DR
LongViTU is a large-scale instruction-tuning dataset and benchmark designed to push Multimodal Language Models (MLMs) toward true long-form video comprehension. By organizing ~900 hours of video into a multi-scale hierarchical tree and applying an LLM-driven self-revision pipeline, the authors generate ~121k open-ended QA pairs with an average contextual reasoning length ("certificate length") of 4.6 minutes. Fine-tuning open-source models on LongViTU significantly narrows the performance gap with commercial systems like Gemini-1.5-Pro and yields substantial gains on external long-video benchmarks like EgoSchema (+30.7%).
What Problem Does This Paper Address?
While vision-language models excel at describing images and short video clips (<60 seconds), long-form video understanding remains a critical bottleneck. Existing automated video question-answering (VQA) datasets suffer from several shortcomings:
- Short temporal scope: Questions rarely require reasoning over long horizons (most average "certificate lengths" are under 60–100 seconds).
- Missing temporal ground truth: QA pairs lack explicit start-and-end event timestamps, making it difficult to pinpoint where models fail.
- Coarse scope and artificial tasks: Datasets often rely on multiple-choice formats, focus purely on static spatial questions (e.g., identifying objects in single frames), or contain verbose, hallucinated answers.
- Context explosion: Directly feeding long video frame sequences into LLMs quickly exceeds token limits and introduces excessive visual redundancy.
Methods
To overcome context limits during dataset generation, LongViTU introduces a three-stage automated synthesis pipeline:
[Frames (1 fps)] ──> InternLM-XComposer2 ──> Frame Captions
│
[Human Annotations (Ego4D)] ───────────────────────┼──> GPT-4 ──> Event Descriptions
│
GPT-4 ──> Segment Summaries
│
[Hierarchical Video Tree]
│
Sliding Window (5 segments)
│
GPT-4 QA Generation (Q: first 3, A: last 2)
│
GPT-4 Self-Revision ──> Final QA
- Hierarchical Video Tree Construction:
- Frame Level: Captions are sampled at 1 fps via
InternLM-XComposer2to capture fine-grained spatial and object details. - Event Level: GPT-4 integrates frame captions with human-annotated event timestamps from Ego4D, eliminating redundancy into coherent event descriptions.
- Segment Level: GPT-4 groups consecutive events into macro-level segments.
- Sliding-Window QA Generation: A sliding window spans 5 segments. To simulate realistic temporal retrospection, GPT-4 poses questions based on events in the first three segments while deriving answers strictly from the subsequent two segments.
- Self-Revision Mechanism: GPT-4 evaluates generated QA pairs against the video context to eliminate hallucinations, strip out giveaway timestamps/indices, and simplify answers into concise, direct ground truths.
- Multi-Level Scoring Criteria: For evaluation, GPT-4 grades open-ended model answers against ground-truth references on a 6-tier rubric (0 to 100).
Datasets
- LongViTU (Introduced): Built on the full Ego4D video suite (~900 hours across 1,833 egocentric videos averaging 29.3 minutes). Contains 121k QA pairs (101k train, 14k val, 6k test benchmark) split across 3 primary categories:
- Spatiotemporal Understanding (55%): Object, Attribute, Location, Action.
- Episodic Reasoning (24.4%): Transition, Interaction, Causality, Motivation.
- Commonsense Inference (20.6%): Planning, Risk, Function, Affordance.
- Evaluation Benchmarks: Evaluated internally on the LongViTU test set and externally on EgoSchema (In-Distribution), and VideoMME, WorldQA, and OpenEQA (Out-of-Distribution).
Key Insights and Ideas
- Hierarchical tree structure beats flat context: Abstracting video from frames to events to segments allows LLMs to retain localized spatial fidelity while understanding long-range event causality, bypassing context-window degradation.
- Fine-grained spatial reasoning is harder than temporal logic: Models scored markedly lower on Spatiotemporal Understanding (Video-LLaVA: 44.9) than on Episodic Reasoning (58.0) and Commonsense Inference (59.8). The sequential nature of LLMs naturally captures narrative logic, but dense spatial localization remains a bottleneck.
- The Blind-QA reality check: Pure text-based GPT-4 without video input achieved an overall score of 38.2. This confirms that text-mediated dataset generation can inherit language priors, emphasizing the need for rigorous multi-modal grounding.
Key Results
- Closing the Gap with Commercial Models: On the LongViTU benchmark, supervised fine-tuning (SFT) boosted Video-LLaVA from 45.9 to 50.7, closely approaching the proprietary Gemini-1.5-Pro (52.3 at 1 fps). LLaMA-VID improved from 38.2 to 44.5 (+16.5%).
- Cross-Benchmark Generalization:
- EgoSchema (ID): SFT Video-LLaVA jumped from 36.8 to 48.1 (+30.7% relative improvement).
- WorldQA (OOD): Increased from 30.2 to 34.1 (+12.9%).
- VideoMME (OOD): Overall improved by +0.6%, with the strongest gains on Long videos (+5.1%) and Medium videos (+6.6%).
Limitations and Future Work
- Text Intermediary Bias: Because dataset generation uses textual frame descriptions as an intermediary, linguistic shortcuts can inadvertently be introduced into the QA pairs.
- Context Utility: Automated generation can occasionally produce QA pairs that are factually accurate but lack deep conversational utility. The authors suggest future iterations will add manual human verification.
- Future Directions: Incorporating streaming video architectures and exploring specialized video memory mechanisms for long-horizon egocentric AI assistants.
Summary updated