Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains

Soumya Rani Samineni, Durgesh Kalwar, Vardaan Gangal, Siddhant Bhambri, Subbarao Kambhampati

TL;DR

Reinforcement Learning with Verifiable Rewards (RLVR) is widely celebrated for boosting LLM performance on reasoning tasks, but because it only rewards final answers, its impact on intermediate steps has remained a black box. This paper introduces "trace coherence"—a logic-based metric to evaluate intermediate reasoning—and reveals that while RL post-training significantly cleans up step-by-step errors, this local polish does not guarantee global logical validity. In short, RLVR makes reasoning chains look much better locally, even when they remain globally incorrect.

What Problem Does This Paper Address?

Post-training LLMs using RLVR (popularized by models like DeepSeek-R1) has yielded massive success in mathematical domains. However, standard RLVR setups use a sparse reward based only on whether the final answer is correct, distributing this reward uniformly across all tokens in the generated sequence. Despite this lack of token-level guidance, claims are frequently made that RLVR inherently improves the model's internal "reasoning traces."

The authors investigate this tension: Does rewarding a final answer actually make the intermediate reasoning steps more logically sound? Because formally verifying the absolute correctness of a mathematical proof ("trace validity") is computationally intractable at scale, the authors introduce trace coherence as a proxy. Trace coherence measures local consistency by tracking the absence of specific logical and arithmetic errors in the intermediate steps.

Methods

The authors design an experimental pipeline using the GRPO (Group Relative Policy Optimization) algorithm to fine-tune a Qwen-2.5-0.5B model.

To systematically evaluate the intermediate reasoning traces, they propose a novel First-Order Logic (FOL)-based error taxonomy consisting of four mutually exclusive categories:

  1. False Premise: Misinterpreting the problem statement or introducing unsupported assumptions.
  2. False Rule: Applying incorrect mathematical logic, wrong formulas, or skipping necessary steps.
  3. Calculator Error: Simple arithmetic mistakes (e.g., $5 \times 6 = 10$).
  4. Format Error: Failing to format the final answer as required (e.g., missing the \boxed{} tag).

They leverage GPT-4o as a judge to translate the model's natural language reasoning traces into formal FOL representations and classify any errors according to this taxonomy.

Datasets

  • GSM8K: A standard benchmark of grade-school math word problems (8.5K problems total: 7.5K training, 1K test) used for RL training and evaluation.
  • Human-Annotated Evaluation Dataset: A newly curated dataset containing 25 responses for each error type and 100 error-free responses (200 total) used to benchmark the GPT-4o judge's error-tagging accuracy.

Key Insights and Ideas

  • Local Coherence $\neq$ Global Validity: The most striking takeaway is that RLVR acts more like a local text editor than a global logician. It successfully trains the model to generate steps that look coherent and error-free, but this local improvement doesn't inherently mean the entire chain of logic is globally sound or leads to the right answer.
  • The "Pattern 01" Boost: The most significant gains in trace coherence occur in scenarios where the base model failed but the RL-trained model succeeded (Pattern 01). RLVR effectively teaches the model to avoid glaring intermediate blunders when it manages to hit the right final answer.
  • The Illusion of Reasoning: Because rewards are distributed uniformly across all tokens, the model learns to optimize for the appearance of structured thinking. It eliminates local contradictions without necessarily understanding the holistic structure of the mathematical proof.

Key Results

  • Judge Accuracy: The GPT-4o judge achieved an overall accuracy of 57.8% on the human-annotated dataset. It performed exceptionally well at spotting calculator errors but had lower recall on format errors due to the complexities of FOL translation.
Figure 1: Evaluation accuracy
  • Trace Coherence Gains: For problems where the RL model successfully corrected a base model failure (Pattern 01), trace coherence skyrocketed from 0% to approximately 85% across Pass@K evaluations.
  • For problems where both the base and RL models were already correct (Pattern 11), RL post-training pushed trace coherence up to 96%, showing a clear refinement of the reasoning steps.
Figure 2: Confusion matrices for patterns 00 & 01 (left) and 10 & 11 (right) at Pass@1.

Limitations and Future Work

  • Judge Reliability: The LLM-as-a-judge framework achieved a modest 57.8% accuracy, particularly struggling with format errors during FOL conversion, indicating room for more robust verification methods.
  • Scale and Model Size: The study was restricted to a small model (Qwen-2.5-0.5B); it remains to be seen if larger models exhibit the same disconnect between local coherence and global validity under RLVR.
  • Future Directions: The authors suggest that future RLVR methods need to move away from uniform token rewards and instead incorporate token-level or step-level verifiers to ensure true logical validity rather than just superficial coherence.

Summary updated

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR)-based post-training of Large Language Models (LLMs) has been shown to improve accuracy on reasoning tasks and continues to attract significant attention. Existing RLVR methods, however, typically treat all tokens uniformly without accounting for token-level advantages. These methods primarily evaluate performance based on final answer correctness or Pass@K accuracy, and yet make claims about RL post-training leading to improved reasoning traces. This motivates our investigation into the effect of RL post-training on intermediate tokens which are not directly incentivized. To study this, we design an experimental setup using the GRPO algorithm with Qwen-2.5-0.5B model on the GSM8K dataset. We introduce trace coherence, a First-Order Logic (FOL)-based measure to capture the consistency of reasoning steps by identifying errors in the traces. We distinguish between trace validity and trace coherence, noting that the former implies logical soundness while the latter measures local coherence via lack of errors. Our results show that RL post-training overall improves trace coherence with the most significant gains on problems where the base model fails but the RL model succeeds. Surprisingly, RL enhances local coherence without necessarily producing valid or correct solutions. This highlights a crucial distinction: improved local coherence in reasoning steps does not guarantee final answer correctness. We argue that claims of improved reasoning via RL must be examined with care, as these may be based on improved trace coherence, which may not translate into fully valid mathematical proofs.
BibTeX
@inproceedings{samineni2025local,
  author = {Soumya Rani Samineni and Durgesh Kalwar and Vardaan Gangal and Siddhant Bhambri and Subbarao Kambhampati},
  title = {{Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains}},
  booktitle = {ER @ NeurIPS},
  year = {2025},
  url = {https://openreview.net/forum?id=d9McZKIT93},
}
Figure