TL;DR
Reinforcement Learning with Verifiable Rewards (RLVR) can inadvertently train video models to "cheat" by relying on textual shortcuts rather than visual evidence. This paper introduces Bifocal Self-distillation Policy Optimization (BSPO), a framework that uses self-generated reasoning principles and visual perturbations to ensure models remain grounded in the video while improving their reasoning capabilities.
What Problem Does This Paper Address?
Large Multimodal Models (LMMs) are increasingly trained using RLVR—a paradigm where models are rewarded simply for getting the right answer (outcome-level optimization). While effective for text, the authors discover a critical failure mode in video reasoning: Shortcut Amplification.
Because many video questions can be guessed using common sense or linguistic patterns (e.g., "If a team scores, the stadium probably cheers"), RLVR often encourages the model to ignore the video and double down on these "textual priors." The authors characterize this as Perceptual Uncertainty Collapse: as training progresses, the model becomes more confident, but that confidence is increasingly decoupled from the actual visual evidence.
Methods
To combat this, the authors propose BSPO, which moves beyond simple "right/wrong" rewards by reweighting the learning process based on two new signals:
- Principle-guided Credit Assignment (PCA): Instead of treating every word in a correct answer as equally important, the model is asked to generate "reasoning principles" by comparing its own successful and failed attempts. It then gives more "credit" to tokens that align with these logical principles.
- Shortcut-aware Contrastive Perturbation (SCP): This is the "grounding" check. The system creates a "counterfactual" version of the video (e.g., by shuffling frames or cropping). If the model’s prediction remains the same even when the visual evidence is destroyed, it’s a sign of shortcut reliance. BSPO penalizes this invariance, forcing the model to be sensitive to the actual video content.
These two signals are combined into a "bifocal" objective that adaptively modulates the RL updates, ensuring they are both logically sound and visually grounded.
Datasets
The models were trained on 20,000 QA pairs sampled from the Video-R1-260k dataset. Evaluation was conducted across six diverse benchmarks to ensure a comprehensive look at video reasoning:
- Long Video: Video-MME and LVBench.
- General Video: TempCompass and MVBench.
- Complex Reasoning: Video-MMMU and MMVU.
The authors also utilized a custom Counterfactual Set, where key video segments were replaced with visually irrelevant but semantically related clips to test for shortcut reliance.
Key Insights and Ideas
- The Entropy Gap: The authors use "visual-conditioning entropy gap" ($\bar{G}$) to measure how much the video actually helps the model. In standard RL, this gap shrinks over time, meaning the video becomes less relevant to the model's decision-making.
- Gradient Misalignment: Through a clever "mechanistic analysis," they show that RL updates often align with directions that reduce uncertainty based on text, while the alignment with visual-based uncertainty reduction actually declines.
- Self-Correction via Principles: The model can act as its own teacher. By summarizing why it got an answer right or wrong, it creates a feedback loop that refines its own internal credit assignment.
Key Results
BSPO consistently outperformed standard RLVR baselines (like GRPO and DAPO) and other self-distillation methods across all benchmarks.
- Performance Gains: On the Qwen3-VL-4B backbone, BSPO achieved a +6.2% improvement on MMVU and +5.7% on TempCompass compared to the base model.
- Visual Faithfulness: Unlike standard RL, which saw a narrowing gap between faithful and counterfactual tests, BSPO actually enlarged this gap. This proves the model became more robust to shortcuts and more dependent on actual visual cues.
Limitations and Future Work
The authors acknowledge that BSPO relies on the model's inherent ability to generate meaningful reasoning principles. For very small or weak models, these self-generated "principles" might be low-quality, potentially limiting the effectiveness of the PCA component.
Future work could explore more sophisticated video perturbation techniques beyond shuffling and cropping, or investigate how to scale this "bifocal" approach to even larger multimodal models where the risk of complex, high-level shortcuts is even greater.
Summary updated