Rectify Evaluation Preference: Improving LLMs' Critique on Math Reasoning via Perplexity-aware Reinforcement Learning

Changyuan Tian, Zhicong Lu, Shuang Qian, Nayu Liu, Peiguang Li, Li Jin, Leiyi Hu, Zhizhao Zeng, Sirui Wang, Ke Zeng, Zhi GuoarXiv 2025 · DOI · arXiv

Abstract

While Large Language Models (LLMs) have become proficient at solving mathematical problems, their ability to critique reasoning—identifying exactly where a multi-step solution fails—remains a significant bottleneck. This paper identifies a fundamental behavioral bias termed "imbalanced evaluation preference," where models reflexively judge solutions with lower perplexity (those matching their own linguistic style) as correct. We propose a perplexity-aware reinforcement learning framework based on Group Relative Policy Optimization (GRPO) that recalibrates these biases. By modulating rewards based on perplexity and implementing class-level loss aggregation, our method forces models to explore "counter-preference" trajectories. Experimental results on the ProcessBench and a new OPS benchmark demonstrate that our approach significantly improves error localization and achieves state-of-the-art performance across diverse reasoning styles.

Introduction

As LLMs are increasingly used for complex reasoning tasks, the need for reliable automated critique becomes paramount. However, current models often fail to catch errors in unfamiliar reasoning chains and unfairly penalize correct solutions that do not align with their internal linguistic distribution. This "narcissistic" evaluation behavior suggests that the challenge in training robust critics is not merely a lack of high-quality data, but a deep-seated stylistic bias.

The Perplexity Bias in LLM Evaluation

Statistical analysis reveals a clear, negative linear correlation between a solution's perplexity and the model's "Balance Indicator" (BI). As a solution becomes more linguistically "alien" to the model (higher perplexity), the model becomes increasingly likely to judge it as "wrong," regardless of its underlying logic. This bias creates a systemic blind spot in cross-model evaluation, where models perform significantly better at critiquing their own outputs than those of other families.

Correlation between perplexity and the Balance Indicator (BI) across Qwen2, Llama-3.1, and Mistral-7B.

Methodology: Perplexity-aware GRPO

To address this, we introduce a novel training approach that enhances standard Group Relative Policy Optimization (GRPO) with two primary components:

  1. Perplexity-aware Modulation: Instead of uniform reward treatment, the algorithm uses perplexity to rescale rewards. Higher weights are assigned to "counter-preference" scenarios—such as correctly identifying a low-perplexity solution as incorrect. This prevents the model from relying on linguistic heuristics.
  2. Class-Level Loss Aggregation: To prevent gradients from being dominated by a single class of labels, we implement a balanced loss function that aggregates "correct" and "wrong" samples independently.
  3. Graded Reward Function: We utilize a hierarchical reward system that combines format compliance with an F1-based answer reward to precisely localize the first erroneous step in a reasoning chain.
Overview of the proposed Perplexity-aware Modulation framework compared to standard GRPO.

Experimental Results

We evaluated our framework using two primary benchmarks: ProcessBench, which assesses error localization across GSM8K and MATH datasets, and OPS (One-to-many Problem-Solution), a new benchmark designed to isolate style-based evaluation changes.

Key Findings

  • Bias Reduction: The proposed method significantly lowered the absolute Balance Indicator (|BI|). For the Qwen2-7B base, |BI| dropped from 34.39 to 17.14, indicating a much more neutral evaluation process.
  • State-of-the-Art Performance: Perplexity-aware GRPO outperformed vanilla GRPO and other variants in 11 out of 15 test cases on ProcessBench.
  • Generalization: Unlike standard RL, which often overfits to specific training distributions, our perplexity-aware approach showed superior generalization across different mathematical datasets.

Conclusion and Future Work

By treating perplexity as a signal for calibration rather than a noise factor, we can significantly improve the objectivity of LLM-based critics. While this study focuses on mathematical reasoning, future work will explore whether similar perplexity biases exist in domains such as code critiquing or legal reasoning. Additionally, reducing the computational overhead of perplexity modulation remains a key area for optimization.

Summary updated

Abstract

To improve Multi-step Mathematical Reasoning (MsMR) of Large Language Models (LLMs), it is crucial to obtain scalable supervision from the corpus by automatically critiquing mistakes in the reasoning process of MsMR and rendering a final verdict of the problem-solution. Most existing methods rely on crafting high-quality supervised fine-tuning demonstrations for critiquing capability enhancement and pay little attention to delving into the underlying reason for the poor critiquing performance of LLMs. In this paper, we orthogonally quantify and investigate the potential reason -- imbalanced evaluation preference, and conduct a statistical preference analysis. Motivated by the analysis of the reason, a novel perplexity-aware reinforcement learning algorithm is proposed to rectify the evaluation preference, elevating the critiquing capability. Specifically, to probe into LLMs' critiquing characteristics, a One-to-many Problem-Solution (OPS) benchmark is meticulously constructed to quantify the behavior difference of LLMs when evaluating the problem solutions generated by itself and others. Then, to investigate the behavior difference in depth, we conduct a statistical preference analysis oriented on perplexity and find an intriguing phenomenon -- ``LLMs incline to judge solutions with lower perplexity as correct'', which is dubbed as \textit{imbalanced evaluation preference}. To rectify this preference, we regard perplexity as the baton in the algorithm of Group Relative Policy Optimization, supporting the LLMs to explore trajectories that judge lower perplexity as wrong and higher perplexity as correct. Extensive experimental results on our built OPS and existing available critic benchmarks demonstrate the validity of our method.
BibTeX
@misc{tian2025rectify,
  author = {Changyuan Tian and Zhicong Lu and Shuang Qian and Nayu Liu and Peiguang Li and Li Jin and Leiyi Hu and Zhizhao Zeng and Sirui Wang and Ke Zeng and Zhi Guo},
  title = {{Rectify Evaluation Preference: Improving LLMs' Critique on Math Reasoning via Perplexity-aware Reinforcement Learning}},
  year = {2025},
  eprint = {2511.10303},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.10303},
}
Figure