VerIPO: Cultivating Long Reasoning in Video-LLMs via Verifier-Gudied Iterative Policy Optimization

Yunxin Li, Xinyu Chen, Zitao Li, Zhenyu Liu, Longyue Wang, Wenhan Luo, Baotian Hu, Min ZhangarXiv 2025 · DOI · arXiv

Introduction

While Large Language Models (LLMs) have achieved significant milestones in complex text-based reasoning, Video-LLMs continue to struggle with multi-step logical inference. Current Reinforcement Fine-Tuning (RFT) methods for video are hampered by expensive data annotation, logical inconsistencies (where models reach the right answer through flawed reasoning), and training instability.

To address these challenges, we introduce VerIPO (Verifier-guided Iterative Policy Optimization). VerIPO is a novel training framework designed to cultivate "thinking" capabilities in Video-LLMs. By integrating the exploratory power of Group Relative Policy Optimization (GRPO) with a specialized Rollout-Aware Verifier and efficient Direct Preference Optimization (DPO), VerIPO enables models to generate long, logically consistent reasoning chains with a 7x speedup in training compared to standard RL methods.

Methodology: The GRPO-Verifier-DPO Loop

The core of VerIPO is an iterative training loop that balances broad exploration with targeted refinement.

Initial Policy Exploration (GRPO)

The model first uses GRPO to explore a wide range of possible reasoning paths for a given video. To maximize flexibility during this discovery phase, the standard KL-divergence constraint is removed, allowing the model to deviate from its initial policy to find more effective reasoning strategies.

Rollout-Aware Verifier

A dedicated "judge" model (Qwen3-8B) evaluates the generated reasoning chains (rollouts) based on four key metrics:

  • Accuracy: Correctness of the final answer.
  • Consistency: Alignment between the internal "thinking" steps and the final output.
  • Repetition: Absence of logical loops.
  • Length: Sufficient detail in the reasoning process.

Policy Refinement (DPO)

The verifier curates preference pairs from the rollouts (e.g., a long, consistent chain vs. a short, inconsistent one). These pairs are then used in a DPO stage. This refinement process is significantly faster than GRPO and is crucial for stabilizing the model's reasoning length and logical flow.

Impact of DPO on training accuracy during Iteration 2.

Experimental Setup

Training follows a multi-stage curriculum designed to activate reasoning before applying it to video:

  1. Reasoning Activation: Initial training on text-only and image-text datasets (e.g., QuALITY, DAPO-Math) provides a "cold start" for logical processing.
  2. Video Reasoning: Fine-tuning on filtered subsets of diverse video benchmarks like MVBench and LongVideoBench.
  3. Iterative Pruning: During training, the verifier discards 80% of simpler examples already mastered by the model, focusing computational resources on challenging cases.

Key Results and Insights

VerIPO demonstrates superior performance and efficiency across multiple dimensions.

State-of-the-Art Performance

Based on the Qwen2.5-VL-7B architecture, VerIPO outperforms much larger models and specialized reasoning models. On the VSI-Bench (spatial reasoning), the 7B VerIPO model achieved 41.8% accuracy, surpassing models like Video-R1 and Kimi-VL.

Performance comparison on VSI-Bench across various models.

Efficiency and Reasoning Quality

The DPO refinement stage proved to be 7x faster per sample than the GRPO exploration stage. Furthermore, on the TOMATO benchmark (temporal reasoning), VerIPO increased the "Acc-True" score (correct answer plus correct logic) from 26.5% to 31.3%, while nearly doubling the average response length from 89 to 202 tokens.

The "Slow Search, Fast Align" Dynamic

A key insight from this work is that while GRPO is effective for discovering new logical paths (slow search), DPO is far superior for reinforcing specific stylistic and logical traits, such as avoiding repetition and ensuring detailed explanations (fast align).

Conclusion

VerIPO offers a robust framework for enhancing the reasoning capabilities of Video-LLMs. By combining iterative exploration with verifier-guided preference optimization, it mitigates common issues like logical inconsistency and training instability. Future work will focus on reducing dependence on the "judge" model's quality and extending these long-reasoning capabilities to other modalities such as audio and robotics control.

Summary updated

Abstract

Applying Reinforcement Learning (RL) to Video Large Language Models (Video-LLMs) shows significant promise for complex video reasoning. However, popular Reinforcement Fine-Tuning (RFT) methods, such as outcome-based Group Relative Policy Optimization (GRPO), are limited by data preparation bottlenecks (e.g., noise or high cost) and exhibit unstable improvements in the quality of long chain-of-thoughts (CoTs) and downstream performance.To address these limitations, we propose VerIPO, a Verifier-guided Iterative Policy Optimization method designed to gradually improve video LLMs' capacity for generating deep, long-term reasoning chains. The core component is Rollout-Aware Verifier, positioned between the GRPO and Direct Preference Optimization (DPO) training phases to form the GRPO-Verifier-DPO training loop. This verifier leverages small LLMs as a judge to assess the reasoning logic of rollouts, enabling the construction of high-quality contrastive data, including reflective and contextually consistent CoTs. These curated preference samples drive the efficient DPO stage (7x faster than GRPO), leading to marked improvements in reasoning chain quality, especially in terms of length and contextual consistency. This training loop benefits from GRPO's expansive search and DPO's targeted optimization. Experimental results demonstrate: 1) Significantly faster and more effective optimization compared to standard GRPO variants, yielding superior performance; 2) Our trained models exceed the direct inference of large-scale instruction-tuned Video-LLMs, producing long and contextually consistent CoTs on diverse video reasoning tasks; and 3) Our model with one iteration outperforms powerful LMMs (e.g., Kimi-VL) and long reasoning models (e.g., Video-R1), highlighting its effectiveness and stability.
BibTeX
@misc{li2025veripo,
  author = {Yunxin Li and Xinyu Chen and Zitao Li and Zhenyu Liu and Longyue Wang and Wenhan Luo and Baotian Hu and Min Zhang},
  title = {{VerIPO: Cultivating Long Reasoning in Video-LLMs via Verifier-Gudied Iterative Policy Optimization}},
  year = {2025},
  eprint = {2505.19000},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.19000},
}
Figure