JudgeLRM: Large Reasoning Models as a Judge

Nuo Chen, Zhiyuan Hu, Zou Qingyun, Jiaying Wu, Qian Wang, Bryan Hooi, Bingsheng HeSubmission to ICLR 2026 · Rejected

TL;DR

Supervised fine-tuning (SFT) often fails when turning LLMs into evaluators for complex tasks because judgment requires deep, multi-step reasoning rather than surface-level pattern imitation. JudgeLRM addresses this by training compact models (3B to 14B parameters) via Group Relative Policy Optimization (GRPO) using tailored, outcome-driven rewards. Without any step-by-step process supervision, JudgeLRM elicits emergent evaluative reasoning, allowing sub-14B models to outperform both GPT-4 and DeepSeek-R1 on human-grounded evaluation benchmarks.

What Problem Does This Paper Address?

The "LLM-as-a-Judge" paradigm offers a scalable alternative to expensive human annotations, but specialized evaluator models traditionally rely on SFT. The authors demonstrate that SFT hits a performance ceiling when evaluation tasks require deeper reasoning: across different domains, SFT performance gains negatively correlate ($R^2 = 0.53$) with the proportion of samples requiring substantive reasoning.

Evaluating open-ended generation is inherently an abductive and verification-heavy task—judges must extract implicit criteria, check factual claims, identify subtle logical flaws, and justify trade-offs. Standard SFT encourages models to mimic superficial rationale styles without internalizing the underlying verification mechanics.

Methods

JudgeLRM reframes judgment training as a reinforcement learning (RL) problem using Group Relative Policy Optimization (GRPO):

  • Group-Wise Advantage Calculation: Eliminates the need for a separate value model and normalizes advantages within query groups, stabilizing policy updates across tasks of wildly varying difficulty.
  • Judge-Wise Outcome Reward Function ($r = R_{\text{struct}} + R_{\text{content}}$):
  • Structural Reward ($R_{\text{struct}}$): Enforces that the model first generates an explicit thinking trace within <think>...</think> tags followed by valid numerical scores within <answer>...</answer> tags.
  • Relation Reward ($r_{\text{relation}}$): The dominant term (+2.0 / -1.5), rewarding correct win/tie/loss pairwise ordering.
  • Absolute Reward ($r_{\text{absolute}}$): A hierarchical reward (+1.0 for exact score matching; +0.6 for minor score deviations within $\le 2$) to calibrate score magnitudes.
  • Discrete Confidence Reward ($r_{\text{confidence}}$): Adds a +0.2 bonus when the predicted score gap matches or exceeds the ground-truth margin. Crucially, this reward is discrete and satisficing (acting like a reverse hinge loss) rather than linear, which prevents reward hacking and score extremization.

Datasets

  • JudgeLM Dataset: Used exclusively for RL training (100k pairs) and in-distribution evaluation (5k pairs), annotated with GPT-4 scores.
  • PandaLM: A 1k test set featuring human-annotated pairwise win/tie/loss preferences across five broad domains (e.g., Office Productivity, Entertainment Media, Search & Information Retrieval), used as an out-of-distribution (OOD) benchmark.
  • Base Models: Evaluated on top of Qwen2.5-Instruct (3B, 7B, 14B) and Qwen3 (4B, 8B).

Key Insights and Ideas

  • Reasoning Demand Correlates with RL Gains: While SFT struggles on reasoning-heavy categories, JudgeLRM shows a strong positive correlation ($R^2 = 0.95$) between performance gains and the reasoning demand of the domain.
  • Emergence of Micro-Cognitive Behaviors: Despite receiving only final outcome rewards (no process supervision), the model spontaneously developed a structured three-step cognitive pipeline: evaluate Assistant 1, evaluate Assistant 2, and execute comparative synthesis with verification, subgoal decomposition, and error spotting.
  • Length Penalization vs. Repetition: Explicitly rewarding response length degraded F1 performance by ~3%, proving that longer output via verbosity/repetition does not substitute for genuine reasoning.
  • Mitigating Positional Bias: Standard LLM judges often favor the first response. JudgeLRM reduced positional bias towards the first candidate from 19.8% (JudgeLM) down to 4.04%–5.39%, achieving up to 89.55% self-consistency under answer-order swapping.

Key Results

  • Outperforming State-of-the-Art LRMs: On the out-of-distribution, human-annotated PandaLM benchmark:
  • JudgeLRM-3B achieved 72.12% F1, beating its teacher GPT-4 (61.80% F1).
  • JudgeLRM-7B (75.05% F1), 8B (75.54% F1), and 14B (76.29% F1) all outperformed DeepSeek-R1 (72.48% F1).
  • Excluding ties (~10% of cases), JudgeLRM-7B reached 83.47% F1, surpassing DeepSeek-R1 (83.38% F1).
  • Dominance Over Baseline Parliaments: JudgeLRM-3B outperformed alternative preference optimization frameworks of identical model size, including Direct Preference Optimization (DPO-Answer: 47.27% F1), Bradley-Terry reward models (58.94% F1), and SFT reasoning-trace distillation (SFT-Distill-R1-Think: 67.33% F1).

Limitations and Future Work

  • Scope of Evaluation: The study focuses on pairwise comparisons in general open-ended QA. It does not thoroughly benchmark single-answer absolute grading or specialized, closed-domain evaluations (e.g., specialized code/math benchmarks like RewardBench or JudgeBench).
  • Safety and Reliability Risks: Because the judge is optimized for alignment with specific reward signals, it still risks inheriting subtle biases from source datasets. The authors explicitly state it is not yet suitable for high-stakes decisions in healthcare, finance, or law without human oversight.

Summary updated

Abstract

Large Language Models (LLMs) are increasingly adopted as evaluators, offering a scalable alternative to human annotation. However, existing supervised fine-tuning (SFT) approaches often fall short in domains that demand complex reasoning. Judgment is inherently reasoning-intensive: beyond surface-level scoring, it requires verifying evidence, identifying errors, and justifying decisions. Through the analysis of evaluation tasks, we find a negative correlation between SFT performance gains and the proportion of reasoning-demanding samples, revealing the limits of SFT in such scenarios. To address this, we introduce JudgeLRM, a family of judgment-oriented LLMs, trained using reinforcement learning (RL) with judge-wise, outcome-driven rewards to activate reasoning capabilities. JudgeLRM consistently outperforms SFT-tuned baselines in the same size, as well as other RL and SFT variants, and even surpass state-of-the-art reasoning models: notably, on the human-generated out-of-distribution PandaLM benchmark, JudgeLRM-3B/4B surpasses its general-purpose teacher GPT-4, while JudgeLRM-7B/8B/14B outperforms DeepSeek-R1 by over 2\% in F1 score, with particularly strong gains on reasoning-heavy tasks. Our findings underscore the value of RL in unlocking reasoning-aligned LLM judges. The code is available at \url{https://anonymous.4open.science/r/JudgeLRM-D1C4/}.
BibTeX
@misc{chen2025judgelrm,
  author = {Nuo Chen and Zhiyuan Hu and Zou Qingyun and Jiaying Wu and Qian Wang and Bryan Hooi and Bingsheng He},
  title = {{JudgeLRM: Large Reasoning Models as a Judge}},
  year = {2025},
  eprint = {2504.00050},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.00050},
}

Loading related content...

Figure