TL;DR
Video-LLMs are great at understanding video but struggle with fine-grained spatial, temporal, and cross-frame details. This paper introduces VideoPASTA, a novel framework that uses targeted adversarial examples and Direct Preference Optimization (DPO) to significantly improve Video-LLM alignment with minimal data (only 7,020 preference pairs). This approach offers a more efficient and effective way to enhance Video-LLM capabilities, outperforming existing methods.
What Problem Does This Paper Address?
Video-Language Models (Video-LLMs) have made impressive strides in understanding video content, but they often falter when it comes to nuanced aspects like spatial relationships between objects, the natural progression of events over time (temporal coherence), and maintaining consistency across different parts of a video (cross-frame continuity). Existing methods to improve these models often rely on massive datasets, extensive pre-training, or complex architectural changes, which are computationally expensive and not always effective. Furthermore, many current approaches focus on general instruction tuning or only address one specific failure mode, leaving core alignment challenges unaddressed. This paper aims to bridge this gap by developing a data-efficient and model-agnostic method to specifically target and correct these critical alignment weaknesses in Video-LLMs.
Methods
The core of VideoPASTA is its novel approach to generating preference data and using it for alignment. Instead of relying on human annotations or proprietary models, VideoPASTA crafts adversarial examples that deliberately violate spatial, temporal, or cross-frame relationships.
Here's a breakdown:
- Targeted Adversarial Sampling: For each video and query, VideoPASTA generates an "aligned" (preferred) response and three "adversarial" responses. These adversarial responses are designed to specifically introduce errors in:
- Spatial Misalignment: Distorting object positions or relationships.
- Temporal Incoherence: Violating the natural order of events.
- Cross-Frame Disconnection: Breaking continuity across distant frames.
- Direct Preference Optimization (DPO): The generated preference pairs (aligned vs. adversarial) are used to fine-tune Video-LLMs using DPO. This technique optimizes the model to prefer the correct responses over the deliberately flawed ones.
- Filtering and Quality Control: A lightweight LLM (Qwen2.5-32B) is used to filter the generated pairs, ensuring that adversarial examples are genuinely incorrect and that aligned responses are accurate. This rigorous filtering process results in a high-quality dataset of only 7,020 preference pairs.
- Model-Agnostic: The framework is designed to be applied to any existing Video-LLM without requiring architectural changes.
Datasets
- Training Data Generation: The VideoPASTA framework generates its own preference pairs. It starts with 3,000 videos from ActivityNet (Yu et al., 2019) for generating queries and responses. After filtering, approximately 7,020 high-quality preference pairs are created.
- Evaluation Benchmarks: The paper evaluates VideoPASTA on a variety of established video understanding benchmarks:
- TempCompass (temporal understanding)
- PerceptionTest (visual perception)
- NeXTQA (compositional reasoning)
- MVBench (multi-task reasoning)
- LongVideoBench (hour-long videos)
- MLVU (multi-task, 3-minute to 2-hour videos)
- VideoMME (6 visual domains, 30 subfields, 11-second to 1-hour videos)
Key Insights and Ideas
- Quality over Quantity: The most striking insight is that a small, carefully curated dataset of adversarial examples (7,020 pairs) can be far more effective than much larger, generic datasets for improving Video-LLM alignment. This challenges the prevailing "bigger is better" paradigm.
- Targeted Alignment is Key: The paper strongly suggests that addressing specific failure modes (spatial, temporal, cross-frame) through targeted adversarial training is more impactful than broad instruction tuning or architectural modifications.
- Self-Improvement Potential: VideoPASTA can even be used for self-improvement, where the target model generates its own preference pairs, demonstrating a powerful loop for continuous learning.
- Generalizability: The framework's effectiveness extends to smaller models and diverse video domains, highlighting its robust and transferable learning signal.
- Efficiency: The method achieves significant gains with minimal data and without human annotation or proprietary models, making it a scalable and practical solution.
Key Results
- Significant Performance Gains: VideoPASTA consistently improves performance across various foundation models and benchmarks. For example, it achieved gains of up to +3.8 percentage points on LongVideoBench, +4.1 on VideoMME, and +4.0 on MVBench.
- Outperforming SOTA: When combined with models like Qwen2.5-VL, VideoPASTA achieved top scores on several benchmarks, surpassing existing state-of-the-art models.
- Efficiency Advantage: VideoPASTA achieved superior results with only 7k pairs compared to TPO (10k pairs) and Hound-DPO (17k pairs), demonstrating a significant efficiency improvement (e.g., ~16x more efficient than TPO on MVBench).
- Qualitative Improvements: The paper provides qualitative examples showing VideoPASTA's ability to correct spatial misalignments, temporal incoherence, and cross-frame disconnections where baseline models fail.
Limitations and Future Work
- Dependence on Generator Models: The quality of generated preference pairs relies on the capabilities of the models used for query generation, response generation, and verification. Potential biases in these models could subtly affect the preference dataset.
- Future Work: The authors suggest exploring better evaluation metrics for model reasoning and further investigating the impact of different prompt designs and adversarial strategies. They also note that future work could explore how to further enhance the efficiency and scalability of the approach.
Summary updated