Confident First, Reflective Second: Reinforcement Learning of Confidence-Preserving Reflection for Video Understanding

Zihao Lin, Jingyuan Liu, Yucheng Zhang, Yuguang Yao, Lifu HuangarXiv 2026 · arXiv

TL;DR

Researchers have identified a "confidence erosion" problem in video-understanding agents: when trained with standard reinforcement learning to revise their answers, models often stop trying to be right the first time, treating their initial response as a throwaway draft. The proposed COFIRE framework fixes this by enforcing a "confident first, reflective second" principle, protecting initial-answer reliability through targeted reward incentives, decision anchoring, and gradient surgery to achieve state-of-the-art results on long-video benchmarks.

What Problem Does This Paper Address?

Modern multimodal large language models (MLLMs) often struggle with long videos because they can only process a fraction of the total frames at once. To compensate, many systems use a "reflection" loop: the model gives an initial answer based on a quick preview, then re-examines specific segments to confirm or revise that answer.

While this sounds intuitive, the authors found that training these loops with standard Reinforcement Learning (RL) leads to confidence erosion. In this context, "confidence" isn't the model's self-reported certainty, but rather the label-verified reliability of its first guess.

Because standard RL typically rewards only the final outcome, the model learns that it doesn't matter if the first answer is wrong, as long as the second one is right. This reduces the marginal value of a correct initial answer. Consequently, the model’s first-pass accuracy stalls, and it becomes overly dependent on the revision step. As shown in Figure 1, a high "reflection gain" (the improvement from first to second answer) often masks a mediocre system that is simply repairing avoidable early mistakes.

Figure 1: Reflection gain is not reflective progress. Video-MME policies by initial-answer accuracy and reflection gain ∆; dashed lines are iso-lines of final ac- curacy. From the same SFT policy, Outcome RL moves up (more gain, same initial answer), while COFIRE moves right to a higher final accuracy. Gray dots: con- figurations from Table 8.

Methods

The authors propose COFIRE (Confident First, Reflective Second), a framework designed to close three "erosion channels" that degrade initial-answer performance:

  1. Initial-Answer Credit (Closing the Incentive Channel): Instead of rewarding only the final outcome, COFIRE uses stage-routed advantages. The initial answer is rewarded for its own correctness. If the initial accuracy falls below a "confidence floor" (set by a reference policy), an adaptive multiplier ($\lambda$) increases the weight of the initial-answer reward, forcing the model to prioritize getting it right immediately.
  2. Verified-Decision Anchoring (Closing the Drift Channel): To prevent the model from "forgetting" how to answer correctly while learning how to retrieve new frames, COFIRE anchors the model to label-verified correct decisions from a reference policy. It uses a selective cross-entropy loss that only reinforces the model when the reference answer is known to be correct.
  3. Asymmetric Gradient Protection (Closing the Interference Channel): Sometimes, the gradient (the direction of learning) for improving reflection conflicts with the gradient for a correct initial answer. COFIRE uses "gradient surgery" to ensure that reflection updates never undo progress made on initial-answer reliability. If the two conflict, the conflicting component of the reflection gradient is discarded.
Figure 3: Overview of COFIRE. Top: G rollouts of one policy answer (Spre), choose how to check (Sgate), and confirm or revise (Srev). Middle: stage-routed advantages penalize the initial answer of an err-then-fix rollout but credit the repair. Bottom: one mechanism per erosion channel: λ rises below the initial-answer accuracy floor (incentive), verified reference answers anchor the decision token (drift), and only the conflicting part of the reflection gradient is removed (interference).

Datasets

The models were trained on a pool of 42,602 multiple-choice questions from the Reflect-R1 pool, which includes short videos (LLaVA-Video, STAR, etc.) and long videos (Panda-70M).

Evaluation was conducted on three major video understanding benchmarks:

  • Video-MME: A comprehensive multi-modal evaluation.
  • MLVU: Focused on multi-task long video understanding.
  • LongVideoBench: A benchmark specifically for long-context interleaved video-language understanding.

Key Insights and Ideas

  • Reflection Gain is a Trap: A model that shows a massive improvement after "thinking twice" might just be a model that is lazy the first time. True progress is measured by the final accuracy, which is highest when the initial answer is already strong.
  • Coupled Capabilities: Answering and reflecting share the same model parameters. Without specific protection, the act of learning when to look again can interfere with the model's ability to see clearly in the first place.
  • Self-Reported Confidence is Unreliable: The authors found that a model's own internal confidence score is a poor signal for deciding when to revise. External, label-verified anchoring is much more effective for stabilizing training.
  • Long Videos Benefit Most: The gains from COFIRE were most pronounced on long-video benchmarks, where a "bad first look" is most costly and hardest to recover from.

Key Results

COFIRE-7B (based on Qwen2.5-VL-7B) achieved a mean accuracy of 67.0% across the three benchmarks, outperforming the strongest compared 7B methods by 1.7 points.

Compared to standard "Outcome RL":

  • Initial-answer accuracy improved by 4.1 points on average.
  • Final accuracy improved by 3.7 points on average.

These results demonstrate that a "stronger initial answer carries through to a stronger final one," rather than the revision process acting as a perfect safety net for a weak start.

Limitations and Future Work

The authors acknowledge several areas for future exploration:

  • Backbone Scaling: The study focused on a single 7B parameter backbone. Testing these principles on larger models (e.g., 72B+) is a natural next step.
  • Loop Variety: The analysis was limited to a specific "answer-and-re-examine" loop. Extending COFIRE to other agentic behaviors, like multi-step temporal search, remains future work.
  • Task Scope: The evaluation was centered on multiple-choice benchmarks. Applying these confidence-preserving techniques to open-ended generation or video editing tasks could reveal new challenges.

Summary updated

Abstract

Video models answer from sampled frames and may miss the decisive moment. A common remedy is to commit to an answer, re-examine the video, and confirm or revise it, a loop now mainly learned with reinforcement learning (RL). We find that RL erodes the initial answer, a phenomenon we term confidence erosion, where confidence means the label-verified reliability of the initial answer instead of self-reported certainty: because errors can be repaired later, a correct initial answer earns less reward, and the initial answer stalls even as the final answer's gain over it rises. We propose the principle confident first, reflective second: the initial answer must stay reliable on its own, above a floor set by a reference policy, and reflection may only add to it. Our method, CoFire, implements this principle with one mechanism for each erosion channel: reward incentives, parameter drift, and gradient interference. CoFire reaches 67.0% mean accuracy on Video-MME, MLVU, and LongVideoBench, 1.7 points above the strongest compared 7B method, and improves initial-answer and final accuracy over outcome-reward RL by 4.1 and 3.7 points on average: a stronger initial answer carries through to a stronger final one.
BibTeX
@misc{lin2026confident,
  author = {Zihao Lin and Jingyuan Liu and Yucheng Zhang and Yuguang Yao and Lifu Huang},
  title = {{Confident First, Reflective Second: Reinforcement Learning of Confidence-Preserving Reflection for Video Understanding}},
  year = {2026},
  eprint = {2610.36467},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36467},
}
Figure