TL;DR
Reinforcement Learning with Verifiable Rewards (RLVR) is widely celebrated for boosting LLM performance on reasoning tasks, but because it only rewards final answers, its impact on intermediate steps has remained a black box. This paper introduces "trace coherence"—a logic-based metric to evaluate intermediate reasoning—and reveals that while RL post-training significantly cleans up step-by-step errors, this local polish does not guarantee global logical validity. In short, RLVR makes reasoning chains look much better locally, even when they remain globally incorrect.
What Problem Does This Paper Address?
Post-training LLMs using RLVR (popularized by models like DeepSeek-R1) has yielded massive success in mathematical domains. However, standard RLVR setups use a sparse reward based only on whether the final answer is correct, distributing this reward uniformly across all tokens in the generated sequence. Despite this lack of token-level guidance, claims are frequently made that RLVR inherently improves the model's internal "reasoning traces."
The authors investigate this tension: Does rewarding a final answer actually make the intermediate reasoning steps more logically sound? Because formally verifying the absolute correctness of a mathematical proof ("trace validity") is computationally intractable at scale, the authors introduce trace coherence as a proxy. Trace coherence measures local consistency by tracking the absence of specific logical and arithmetic errors in the intermediate steps.
Methods
The authors design an experimental pipeline using the GRPO (Group Relative Policy Optimization) algorithm to fine-tune a Qwen-2.5-0.5B model.
To systematically evaluate the intermediate reasoning traces, they propose a novel First-Order Logic (FOL)-based error taxonomy consisting of four mutually exclusive categories:
- False Premise: Misinterpreting the problem statement or introducing unsupported assumptions.
- False Rule: Applying incorrect mathematical logic, wrong formulas, or skipping necessary steps.
- Calculator Error: Simple arithmetic mistakes (e.g., $5 \times 6 = 10$).
- Format Error: Failing to format the final answer as required (e.g., missing the
\boxed{}tag).
They leverage GPT-4o as a judge to translate the model's natural language reasoning traces into formal FOL representations and classify any errors according to this taxonomy.
Datasets
- GSM8K: A standard benchmark of grade-school math word problems (8.5K problems total: 7.5K training, 1K test) used for RL training and evaluation.
- Human-Annotated Evaluation Dataset: A newly curated dataset containing 25 responses for each error type and 100 error-free responses (200 total) used to benchmark the GPT-4o judge's error-tagging accuracy.
Key Insights and Ideas
- Local Coherence $\neq$ Global Validity: The most striking takeaway is that RLVR acts more like a local text editor than a global logician. It successfully trains the model to generate steps that look coherent and error-free, but this local improvement doesn't inherently mean the entire chain of logic is globally sound or leads to the right answer.
- The "Pattern 01" Boost: The most significant gains in trace coherence occur in scenarios where the base model failed but the RL-trained model succeeded (Pattern 01). RLVR effectively teaches the model to avoid glaring intermediate blunders when it manages to hit the right final answer.
- The Illusion of Reasoning: Because rewards are distributed uniformly across all tokens, the model learns to optimize for the appearance of structured thinking. It eliminates local contradictions without necessarily understanding the holistic structure of the mathematical proof.
Key Results
- Judge Accuracy: The GPT-4o judge achieved an overall accuracy of 57.8% on the human-annotated dataset. It performed exceptionally well at spotting calculator errors but had lower recall on format errors due to the complexities of FOL translation.
- Trace Coherence Gains: For problems where the RL model successfully corrected a base model failure (Pattern 01), trace coherence skyrocketed from 0% to approximately 85% across Pass@K evaluations.
- For problems where both the base and RL models were already correct (Pattern 11), RL post-training pushed trace coherence up to 96%, showing a clear refinement of the reasoning steps.
Limitations and Future Work
- Judge Reliability: The LLM-as-a-judge framework achieved a modest 57.8% accuracy, particularly struggling with format errors during FOL conversion, indicating room for more robust verification methods.
- Scale and Model Size: The study was restricted to a small model (Qwen-2.5-0.5B); it remains to be seen if larger models exhibit the same disconnect between local coherence and global validity under RLVR.
- Future Directions: The authors suggest that future RLVR methods need to move away from uniform token rewards and instead incorporate token-level or step-level verifiers to ensure true logical validity rather than just superficial coherence.
Summary updated