TL;DR
Researchers have identified a "confidence erosion" problem in video-understanding agents: when trained with standard reinforcement learning to revise their answers, models often stop trying to be right the first time, treating their initial response as a throwaway draft. The proposed COFIRE framework fixes this by enforcing a "confident first, reflective second" principle, protecting initial-answer reliability through targeted reward incentives, decision anchoring, and gradient surgery to achieve state-of-the-art results on long-video benchmarks.
What Problem Does This Paper Address?
Modern multimodal large language models (MLLMs) often struggle with long videos because they can only process a fraction of the total frames at once. To compensate, many systems use a "reflection" loop: the model gives an initial answer based on a quick preview, then re-examines specific segments to confirm or revise that answer.
While this sounds intuitive, the authors found that training these loops with standard Reinforcement Learning (RL) leads to confidence erosion. In this context, "confidence" isn't the model's self-reported certainty, but rather the label-verified reliability of its first guess.
Because standard RL typically rewards only the final outcome, the model learns that it doesn't matter if the first answer is wrong, as long as the second one is right. This reduces the marginal value of a correct initial answer. Consequently, the model’s first-pass accuracy stalls, and it becomes overly dependent on the revision step. As shown in Figure 1, a high "reflection gain" (the improvement from first to second answer) often masks a mediocre system that is simply repairing avoidable early mistakes.
Methods
The authors propose COFIRE (Confident First, Reflective Second), a framework designed to close three "erosion channels" that degrade initial-answer performance:
- Initial-Answer Credit (Closing the Incentive Channel): Instead of rewarding only the final outcome, COFIRE uses stage-routed advantages. The initial answer is rewarded for its own correctness. If the initial accuracy falls below a "confidence floor" (set by a reference policy), an adaptive multiplier ($\lambda$) increases the weight of the initial-answer reward, forcing the model to prioritize getting it right immediately.
- Verified-Decision Anchoring (Closing the Drift Channel): To prevent the model from "forgetting" how to answer correctly while learning how to retrieve new frames, COFIRE anchors the model to label-verified correct decisions from a reference policy. It uses a selective cross-entropy loss that only reinforces the model when the reference answer is known to be correct.
- Asymmetric Gradient Protection (Closing the Interference Channel): Sometimes, the gradient (the direction of learning) for improving reflection conflicts with the gradient for a correct initial answer. COFIRE uses "gradient surgery" to ensure that reflection updates never undo progress made on initial-answer reliability. If the two conflict, the conflicting component of the reflection gradient is discarded.
Datasets
The models were trained on a pool of 42,602 multiple-choice questions from the Reflect-R1 pool, which includes short videos (LLaVA-Video, STAR, etc.) and long videos (Panda-70M).
Evaluation was conducted on three major video understanding benchmarks:
- Video-MME: A comprehensive multi-modal evaluation.
- MLVU: Focused on multi-task long video understanding.
- LongVideoBench: A benchmark specifically for long-context interleaved video-language understanding.
Key Insights and Ideas
- Reflection Gain is a Trap: A model that shows a massive improvement after "thinking twice" might just be a model that is lazy the first time. True progress is measured by the final accuracy, which is highest when the initial answer is already strong.
- Coupled Capabilities: Answering and reflecting share the same model parameters. Without specific protection, the act of learning when to look again can interfere with the model's ability to see clearly in the first place.
- Self-Reported Confidence is Unreliable: The authors found that a model's own internal confidence score is a poor signal for deciding when to revise. External, label-verified anchoring is much more effective for stabilizing training.
- Long Videos Benefit Most: The gains from COFIRE were most pronounced on long-video benchmarks, where a "bad first look" is most costly and hardest to recover from.
Key Results
COFIRE-7B (based on Qwen2.5-VL-7B) achieved a mean accuracy of 67.0% across the three benchmarks, outperforming the strongest compared 7B methods by 1.7 points.
Compared to standard "Outcome RL":
- Initial-answer accuracy improved by 4.1 points on average.
- Final accuracy improved by 3.7 points on average.
These results demonstrate that a "stronger initial answer carries through to a stronger final one," rather than the revision process acting as a perfect safety net for a weak start.
Limitations and Future Work
The authors acknowledge several areas for future exploration:
- Backbone Scaling: The study focused on a single 7B parameter backbone. Testing these principles on larger models (e.g., 72B+) is a natural next step.
- Loop Variety: The analysis was limited to a specific "answer-and-re-examine" loop. Extending COFIRE to other agentic behaviors, like multi-step temporal search, remains future work.
- Task Scope: The evaluation was centered on multiple-choice benchmarks. Applying these confidence-preserving techniques to open-ended generation or video editing tasks could reveal new challenges.
Summary updated