Introduction
While Large Language Models (LLMs) have achieved significant milestones in complex text-based reasoning, Video-LLMs continue to struggle with multi-step logical inference. Current Reinforcement Fine-Tuning (RFT) methods for video are hampered by expensive data annotation, logical inconsistencies (where models reach the right answer through flawed reasoning), and training instability.
To address these challenges, we introduce VerIPO (Verifier-guided Iterative Policy Optimization). VerIPO is a novel training framework designed to cultivate "thinking" capabilities in Video-LLMs. By integrating the exploratory power of Group Relative Policy Optimization (GRPO) with a specialized Rollout-Aware Verifier and efficient Direct Preference Optimization (DPO), VerIPO enables models to generate long, logically consistent reasoning chains with a 7x speedup in training compared to standard RL methods.
Methodology: The GRPO-Verifier-DPO Loop
The core of VerIPO is an iterative training loop that balances broad exploration with targeted refinement.
Initial Policy Exploration (GRPO)
The model first uses GRPO to explore a wide range of possible reasoning paths for a given video. To maximize flexibility during this discovery phase, the standard KL-divergence constraint is removed, allowing the model to deviate from its initial policy to find more effective reasoning strategies.
Rollout-Aware Verifier
A dedicated "judge" model (Qwen3-8B) evaluates the generated reasoning chains (rollouts) based on four key metrics:
- Accuracy: Correctness of the final answer.
- Consistency: Alignment between the internal "thinking" steps and the final output.
- Repetition: Absence of logical loops.
- Length: Sufficient detail in the reasoning process.
Policy Refinement (DPO)
The verifier curates preference pairs from the rollouts (e.g., a long, consistent chain vs. a short, inconsistent one). These pairs are then used in a DPO stage. This refinement process is significantly faster than GRPO and is crucial for stabilizing the model's reasoning length and logical flow.
Experimental Setup
Training follows a multi-stage curriculum designed to activate reasoning before applying it to video:
- Reasoning Activation: Initial training on text-only and image-text datasets (e.g., QuALITY, DAPO-Math) provides a "cold start" for logical processing.
- Video Reasoning: Fine-tuning on filtered subsets of diverse video benchmarks like MVBench and LongVideoBench.
- Iterative Pruning: During training, the verifier discards 80% of simpler examples already mastered by the model, focusing computational resources on challenging cases.
Key Results and Insights
VerIPO demonstrates superior performance and efficiency across multiple dimensions.
State-of-the-Art Performance
Based on the Qwen2.5-VL-7B architecture, VerIPO outperforms much larger models and specialized reasoning models. On the VSI-Bench (spatial reasoning), the 7B VerIPO model achieved 41.8% accuracy, surpassing models like Video-R1 and Kimi-VL.
Efficiency and Reasoning Quality
The DPO refinement stage proved to be 7x faster per sample than the GRPO exploration stage. Furthermore, on the TOMATO benchmark (temporal reasoning), VerIPO increased the "Acc-True" score (correct answer plus correct logic) from 26.5% to 31.3%, while nearly doubling the average response length from 89 to 202 tokens.
The "Slow Search, Fast Align" Dynamic
A key insight from this work is that while GRPO is effective for discovering new logical paths (slow search), DPO is far superior for reinforcing specific stylistic and logical traits, such as avoiding repetition and ensuring detailed explanations (fast align).
Conclusion
VerIPO offers a robust framework for enhancing the reasoning capabilities of Video-LLMs. By combining iterative exploration with verifier-guided preference optimization, it mitigates common issues like logical inconsistency and training instability. Future work will focus on reducing dependence on the "judge" model's quality and extending these long-reasoning capabilities to other modalities such as audio and robotics control.
Summary updated