Video-STaR: Self-Training Enables Video Instruction Tuning with Any Supervision

Orr Zohar, Xiaohan Wang, Yonatan Bitton, Idan Szpektor, Serena Yeung-Levy

Introduction

Large Vision-Language Models (LVLMs) have shown remarkable capabilities in image-based tasks, yet video understanding remains a significant challenge. A primary bottleneck is the scarcity of high-quality video instruction-tuning data. While image datasets are vast, the current state-of-the-art video instruction dataset, VideoInstruct-100K, contains only 13,000 unique videos. Furthermore, existing video QA data often lacks temporal depth and expert-level reasoning, focusing instead on simple descriptions.

Conversely, the computer vision community possesses extensive labeled datasets for tasks like action recognition (Kinetics), temporal localization (STAR), and action quality assessment (FineDiving). However, these labels are typically "raw" (e.g., a single class name or a numerical score) and not in the conversational format required for instruction tuning. Video-STaR (Video Self-Training with augmented Reasoning) addresses this by converting these diverse labels into rich, reasoned instruction-tuning data through a self-training loop.

Methodology: The Video-STaR Framework

Video-STaR employs a "generate-verify-rationalize" cycle to iteratively improve the model's reasoning capabilities using any labeled video dataset.

Overview of the Video-STaR pipeline, illustrating the cycle of answer generation, label verification, and rationalization.

The framework consists of four main stages:

  1. Answer Generation: The model generates an answer and a step-by-step Chain-of-Thought (CoT) rationale for a given video and question.
  2. Label Verification: The generated answer is checked against the ground-truth "gold" label. If the label is missing or incorrect, the response is filtered out.
  3. Label Rationalization: For instances where the model fails to generate the correct label, it is provided with the gold label and prompted to "reason backward" to explain why that label is correct. This allows the model to learn from its failures.
  4. Instruction Tuning: The model is fine-tuned on the verified and rationalized data, and the cycle repeats.

Data and Evaluation

The authors curated VSTaR-1M, a dataset of 1 million video instruction-tuning pairs derived from Kinetics700, STAR-benchmark, and FineDiving. Evaluation was conducted on standard benchmarks including MSVD-QA, MSRVTT-QA, and TempCompass to assess general video understanding and temporal reasoning.

Key Results and Insights

Video-STaR demonstrates that weak supervision (simple labels) can serve as an effective guardrail for training complex reasoning. A critical finding was the efficacy of Label Rationalization. In the FineDiving dataset, the model initially struggled to generate correct scores. By rationalizing known labels, it eventually learned to perform complex sports judging independently.

Qualitative example of the self-training cycle on a diving video, showing improvement from Cycle 0 to Cycle 2.

Key performance gains include:

  • Temporal Understanding: A 10% improvement on the TempCompass benchmark.
  • Task Adaptation: Accuracy increases of 20% on Kinetics700-QA and 15% on FineDiving (AQA).
  • Zero-Shot Capability: Achieving 71.3% on MSVD-QA, rivaling much larger proprietary models.

Conclusion and Future Work

Video-STaR successfully bridges the gap between traditional CV labels and modern instruction tuning. By leveraging self-training, it enables the use of diverse, specialized datasets to enhance LVLM reasoning. Future work will address computational costs and the potential for hallucinations in generated rationales, potentially using more advanced grounded VLMs as verifiers.

Summary updated

BibTeX
@inproceedings{zohar2025video,
  author = {Orr Zohar and Xiaohan Wang and Yonatan Bitton and Idan Szpektor and Serena Yeung-Levy},
  title = {{Video-STaR: Self-Training Enables Video Instruction Tuning with Any Supervision}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=JYV2hrtFSv},
}
Figure