Video DataFlywheel: Resolving the Impossible Data Trinity in Video-Language Understanding

Xiao Wang, Jianlong Wu, Zijia Lin, Fuzheng Zhang, Di Zhang, Liqiang Nie

Abstract

Video-Language Understanding (VLU) relies heavily on high-quality, large-scale datasets. However, existing datasets often struggle to simultaneously satisfy the "Impossible Data Trinity": high quantity, broad diversity, and high annotation quality. We introduce Video DataFlywheel (VidDF), a framework that breaks this bottleneck through iterative model-in-the-loop refinement and a novel distribution-agnostic noise-control method, AdaTaiLr. Our approach enables models to improve their own training data, leading to significant performance gains across video question answering and retrieval benchmarks, even outperforming models with significantly more parameters.

Introduction: The Impossible Data Trinity

In the field of Video-Language Understanding, data is the primary driver of performance. However, researchers face a persistent challenge known as the Impossible Data Trinity. To be effective, a dataset needs three things: Quantity (millions of samples), Diversity (wide range of visual styles), and Quality (accurate textual descriptions).

Existing datasets typically sacrifice one of these pillars. Human-annotated datasets offer high quality but are difficult to scale. Conversely, Automatic Speech Recognition (ASR) datasets provide massive quantity and diversity but are plagued by noise, as spoken words often do not describe the visual content. While AI-driven captioning has been used to refine these datasets, performance often plateaus as models begin to learn from their own hallucinations.

Visualizing the 'Impossible Data Trinity': The proposed approach (red) achieves a superior balance of quantity, diversity, and quality compared to existing datasets.

The Video DataFlywheel (VidDF) Framework

To overcome the scalability plateau, we propose the Video DataFlywheel (VidDF). This framework treats data refinement as a continuous, iterative process rather than a one-off cleaning step. The cycle begins by using existing LLMs and VLMs to generate initial refined captions for a massive ASR dataset. A base model is trained on this data and then fine-tuned on high-quality human data.

Crucially, this improved model is then used to re-annotate the original noisy dataset. Because the model has become more proficient at understanding video, the new captions are of higher quality. This creates a "flywheel" effect where better models produce better data, which in turn trains even more capable models.

The Video DataFlywheel iterative framework (left) and its scalability performance (right) demonstrating sustained gains over baseline methods.

AdaTaiLr: Robust Noise Control

Iterative refinement alone is insufficient if the model cannot distinguish between high-quality synthetic captions and hallucinations. We introduce AdaTaiLr (Adaptive Total Variation Distance Loss), a distribution-agnostic noise-control method. Unlike traditional KL Divergence, which is highly sensitive to outliers, AdaTaiLr utilizes Total Variation Distance (TVD) to encourage sparse solutions, effectively down-weighting noisy or irrelevant captions. By adaptively adjusting the trade-off between bias and variance during training, AdaTaiLr ensures that the model remains robust even as the synthetic dataset grows.

Key Insights and Results

Our evaluation across standard benchmarks (MSR-VTT, MSVD, ActivityNet, etc.) demonstrates the efficacy of the VidDF framework:

  • Performance Boost: VidDF achieved a 3.1% improvement over previous state-of-the-art data refinement methods.
  • Efficiency: Our 7B parameter model outperformed 13B and even 34B parameter models (such as LLaVA-NEXT) on several VideoQA tasks.
  • Retrieval Gains: We observed a 5-6% average improvement in text-to-video retrieval tasks across all benchmarks.
  • The Scalability Paradox: Quantitative analysis confirms that while simple synthetic data addition eventually harms performance, the combination of iterative refinement and AdaTaiLr maintains a positive scaling trajectory.

Furthermore, the authors observed that synthetic captions exhibit a significantly longer "tail" in token frequency than human-annotated text. This explains why models require larger datasets to generalize effectively to rare tokens, highlighting the critical importance of the quantity pillar in the trinity.

Conclusion

Video DataFlywheel provides a scalable solution to the data quality bottleneck in VLU. By combining iterative model-in-the-loop refinement with robust noise control, we demonstrate that models can effectively "bootstrap" their way to higher performance. Future work will explore active data selection and the integration of human feedback to further accelerate the flywheel.

Summary updated

BibTeX
@article{wang2025video,
  author = {Xiao Wang and Jianlong Wu and Zijia Lin and Fuzheng Zhang and Di Zhang and Liqiang Nie},
  title = {{Video DataFlywheel: Resolving the Impossible Data Trinity in Video-Language Understanding}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2025},
  doi = {10.1109/tpami.2025.3528394},
}
Figure