Abstract
While Large Language Models (LLMs) have become proficient at solving mathematical problems, their ability to critique reasoning—identifying exactly where a multi-step solution fails—remains a significant bottleneck. This paper identifies a fundamental behavioral bias termed "imbalanced evaluation preference," where models reflexively judge solutions with lower perplexity (those matching their own linguistic style) as correct. We propose a perplexity-aware reinforcement learning framework based on Group Relative Policy Optimization (GRPO) that recalibrates these biases. By modulating rewards based on perplexity and implementing class-level loss aggregation, our method forces models to explore "counter-preference" trajectories. Experimental results on the ProcessBench and a new OPS benchmark demonstrate that our approach significantly improves error localization and achieves state-of-the-art performance across diverse reasoning styles.
Introduction
As LLMs are increasingly used for complex reasoning tasks, the need for reliable automated critique becomes paramount. However, current models often fail to catch errors in unfamiliar reasoning chains and unfairly penalize correct solutions that do not align with their internal linguistic distribution. This "narcissistic" evaluation behavior suggests that the challenge in training robust critics is not merely a lack of high-quality data, but a deep-seated stylistic bias.
The Perplexity Bias in LLM Evaluation
Statistical analysis reveals a clear, negative linear correlation between a solution's perplexity and the model's "Balance Indicator" (BI). As a solution becomes more linguistically "alien" to the model (higher perplexity), the model becomes increasingly likely to judge it as "wrong," regardless of its underlying logic. This bias creates a systemic blind spot in cross-model evaluation, where models perform significantly better at critiquing their own outputs than those of other families.
Methodology: Perplexity-aware GRPO
To address this, we introduce a novel training approach that enhances standard Group Relative Policy Optimization (GRPO) with two primary components:
- Perplexity-aware Modulation: Instead of uniform reward treatment, the algorithm uses perplexity to rescale rewards. Higher weights are assigned to "counter-preference" scenarios—such as correctly identifying a low-perplexity solution as incorrect. This prevents the model from relying on linguistic heuristics.
- Class-Level Loss Aggregation: To prevent gradients from being dominated by a single class of labels, we implement a balanced loss function that aggregates "correct" and "wrong" samples independently.
- Graded Reward Function: We utilize a hierarchical reward system that combines format compliance with an F1-based answer reward to precisely localize the first erroneous step in a reasoning chain.
Experimental Results
We evaluated our framework using two primary benchmarks: ProcessBench, which assesses error localization across GSM8K and MATH datasets, and OPS (One-to-many Problem-Solution), a new benchmark designed to isolate style-based evaluation changes.
Key Findings
- Bias Reduction: The proposed method significantly lowered the absolute Balance Indicator (|BI|). For the Qwen2-7B base, |BI| dropped from 34.39 to 17.14, indicating a much more neutral evaluation process.
- State-of-the-Art Performance: Perplexity-aware GRPO outperformed vanilla GRPO and other variants in 11 out of 15 test cases on ProcessBench.
- Generalization: Unlike standard RL, which often overfits to specific training distributions, our perplexity-aware approach showed superior generalization across different mathematical datasets.
Conclusion and Future Work
By treating perplexity as a signal for calibration rather than a noise factor, we can significantly improve the objectivity of LLM-based critics. While this study focuses on mathematical reasoning, future work will explore whether similar perplexity biases exist in domains such as code critiquing or legal reasoning. Additionally, reducing the computational overhead of perplexity modulation remains a key area for optimization.
Summary updated