Abstract
Video-Language Understanding (VLU) relies heavily on high-quality, large-scale datasets. However, existing datasets often struggle to simultaneously satisfy the "Impossible Data Trinity": high quantity, broad diversity, and high annotation quality. We introduce Video DataFlywheel (VidDF), a framework that breaks this bottleneck through iterative model-in-the-loop refinement and a novel distribution-agnostic noise-control method, AdaTaiLr. Our approach enables models to improve their own training data, leading to significant performance gains across video question answering and retrieval benchmarks, even outperforming models with significantly more parameters.
Introduction: The Impossible Data Trinity
In the field of Video-Language Understanding, data is the primary driver of performance. However, researchers face a persistent challenge known as the Impossible Data Trinity. To be effective, a dataset needs three things: Quantity (millions of samples), Diversity (wide range of visual styles), and Quality (accurate textual descriptions).
Existing datasets typically sacrifice one of these pillars. Human-annotated datasets offer high quality but are difficult to scale. Conversely, Automatic Speech Recognition (ASR) datasets provide massive quantity and diversity but are plagued by noise, as spoken words often do not describe the visual content. While AI-driven captioning has been used to refine these datasets, performance often plateaus as models begin to learn from their own hallucinations.
The Video DataFlywheel (VidDF) Framework
To overcome the scalability plateau, we propose the Video DataFlywheel (VidDF). This framework treats data refinement as a continuous, iterative process rather than a one-off cleaning step. The cycle begins by using existing LLMs and VLMs to generate initial refined captions for a massive ASR dataset. A base model is trained on this data and then fine-tuned on high-quality human data.
Crucially, this improved model is then used to re-annotate the original noisy dataset. Because the model has become more proficient at understanding video, the new captions are of higher quality. This creates a "flywheel" effect where better models produce better data, which in turn trains even more capable models.
AdaTaiLr: Robust Noise Control
Iterative refinement alone is insufficient if the model cannot distinguish between high-quality synthetic captions and hallucinations. We introduce AdaTaiLr (Adaptive Total Variation Distance Loss), a distribution-agnostic noise-control method. Unlike traditional KL Divergence, which is highly sensitive to outliers, AdaTaiLr utilizes Total Variation Distance (TVD) to encourage sparse solutions, effectively down-weighting noisy or irrelevant captions. By adaptively adjusting the trade-off between bias and variance during training, AdaTaiLr ensures that the model remains robust even as the synthetic dataset grows.
Key Insights and Results
Our evaluation across standard benchmarks (MSR-VTT, MSVD, ActivityNet, etc.) demonstrates the efficacy of the VidDF framework:
- Performance Boost: VidDF achieved a 3.1% improvement over previous state-of-the-art data refinement methods.
- Efficiency: Our 7B parameter model outperformed 13B and even 34B parameter models (such as LLaVA-NEXT) on several VideoQA tasks.
- Retrieval Gains: We observed a 5-6% average improvement in text-to-video retrieval tasks across all benchmarks.
- The Scalability Paradox: Quantitative analysis confirms that while simple synthetic data addition eventually harms performance, the combination of iterative refinement and AdaTaiLr maintains a positive scaling trajectory.
Furthermore, the authors observed that synthetic captions exhibit a significantly longer "tail" in token frequency than human-annotated text. This explains why models require larger datasets to generalize effectively to rare tokens, highlighting the critical importance of the quantity pillar in the trinity.
Conclusion
Video DataFlywheel provides a scalable solution to the data quality bottleneck in VLU. By combining iterative model-in-the-loop refinement with robust noise control, we demonstrate that models can effectively "bootstrap" their way to higher performance. Future work will explore active data selection and the integration of human feedback to further accelerate the flywheel.
Summary updated