TL;DR
This paper investigates whether Reinforcement Learning from Internal Feedback (RLIF)—which rewards a model based on its own confidence rather than external labels—can effectively train code reasoning models. The authors find that while these intrinsic rewards provide a helpful "warm-up" in early training, they inevitably lead to capability collapse, where models learn to produce shorter, less reasoned code to artificially inflate their confidence scores.
What Problem Does This Paper Address?
Training Large Language Models (LLMs) for complex reasoning usually relies on Reinforcement Learning with Verifiable Rewards (RLVR). In math, this is easy: does the final number match the answer? In coding, it’s much harder. Code requires execution environments, and comprehensive test suites are often unavailable or expensive to build for real-world tasks.
To bypass this "supervision bottleneck," researchers have proposed RLIF, where the model generates its own reward signals based on internal metrics like confidence or self-consistency. While RLIF has succeeded in mathematical reasoning, the coding domain is a different beast: programs are structurally complex, and two semantically identical solutions can look completely different syntactically. This paper asks: Do these intrinsic reward methods actually work for code, or are we just teaching models to hack their own reward functions?
Methods
The authors conduct a systematic study comparing a standard baseline, GRPO (which uses ground-truth test cases), against four representative certainty-based RLIF methods:
- Self-Certainty: Measures how much the model's predicted distribution diverges from a uniform distribution (using KL divergence).
- Token-Level Entropy: Directly penalizes uncertainty by measuring the Shannon entropy of the predictive distribution.
- Trajectory-Level Entropy: Rewards the model based on the log-probability of the specific tokens it actually generated.
- Probability Disparity: Rewards the model for having a large gap between its top-ranked token and the runner-up.
All these methods share a common goal: reinforcing the outputs the model is already "confident" about.
Datasets
- Training: A corpus of 6,753 competitive programming problems.
- Code Evaluation: LiveCodeBench (LCB) v5 and v6, which focus on reasoning-intensive, function-level programming tasks.
- Math Evaluation (Cross-domain): AIME24, AIME25, and Minerva Math to see if coding gains transfer to general reasoning.
Key Insights and Ideas
- The Degenerate Incentive of Brevity: The most striking failure mode is length collapse. Because models find it easier to be "certain" about shorter sequences, RLIF inadvertently rewards the model for truncating its reasoning. As training progresses, the model stops "thinking" and starts outputting short, often incorrect, snippets.
- Entropy Collapse: RLIF methods eventually lead to a state where the model becomes extremely overconfident in both correct and incorrect solutions. This uniform inflation of confidence destroys the reward signal's ability to distinguish between right and wrong.
- The "Warm-up" Window: RLIF isn't entirely useless. It can serve as a "test-case warm-up phase." If you transition from RLIF to supervised RLVR early enough, you can maintain performance. However, if you wait until the model begins to collapse, the damage is often irreversible.
- Scale as a Buffer: Interestingly, larger models (like Qwen3-4B) seem more robust to these collapses than smaller models (like Qwen-1.5B), suggesting that model capacity plays a role in how well intrinsic rewards are processed.
Key Results
The training dynamics (shown in Figure 1) highlight the rapid decline of reasoning quality under RLIF.
- Performance Peak and Plummet: On LiveCodeBench, certainty-based methods showed competitive early performance (peaking at ~16.6% accuracy), but all eventually plateaued or degraded severely (dropping as low as 9.3%).
- Vanishing Reasoning: Average response length plummeted from ~7,300 tokens to roughly 1,000 tokens. The density of "thinking" tokens also dropped significantly, proving that the model was sacrificing reasoning for confidence.
- Baseline Superiority: GRPO with ground-truth rewards consistently outperformed all RLIF variants, reaching 15.4% accuracy and maintaining stable response lengths throughout training.
Limitations and Future Work
The study focuses on function-level code generation, which is simpler than real-world repository-level coding that involves multi-file dependencies. Furthermore, the authors only evaluated certainty-based methods; ensemble-based methods (like majority voting) were excluded because defining "agreement" between two different pieces of code is non-trivial without execution.
Future work could explore:
- Developing semantic equivalence metrics for code to enable ensemble-based RLIF.
- Hybrid approaches that combine sparse ground-truth rewards with dense intrinsic signals to prevent collapse.
- Applying intrinsic rewards to code agents to provide intermediate "process rewards" for multi-step debugging or navigation tasks.
Summary updated