Abstract
Large Language Models (LLMs) are increasingly adopted as evaluators, offering a scalable alternative to human annotation. However, existing supervised fine-tuning (SFT) approaches often fall short in domains that demand complex reasoning. Judgment is inherently reasoning-intensive: beyond surface-level scoring, it requires verifying evidence, identifying errors, and justifying decisions. Through the analysis of evaluation tasks, we find a negative correlation between SFT performance gains and the proportion of reasoning-demanding samples, revealing the limits of SFT in such scenarios. To address this, we introduce JudgeLRM, a family of judgment-oriented LLMs, trained using reinforcement learning (RL) with judge-wise, outcome-driven rewards to activate reasoning capabilities. JudgeLRM consistently outperform SFT-tuned baselines in the same size, as well as other RL and SFT variants, and even surpass state-of-the-art reasoning models: notably, JudgeLRM-3B/4B exceeds GPT-4, while JudgeLRM-7B/8B/14B outperforms DeepSeek-R1 by over 2% in F1 score, with particularly strong gains on reasoning-heavy tasks. Our findings underscore the value of RL in unlocking reasoning-aligned LLM judges.
Introduction
The "LLM-as-a-Judge" paradigm—using a powerful model to evaluate the outputs of others—is a cornerstone of modern AI development. However, relying on proprietary models like GPT-4 is expensive and creates a closed-loop dependency. While open-source alternatives exist, they are typically trained via Supervised Fine-Tuning (SFT) to mimic human or GPT-4 scores. This research identifies a critical flaw in the SFT approach: its performance drops sharply as the "reasoning depth" of the evaluation task increases.
The SFT Performance Ceiling
Analysis across various domains—including Office Productivity, Search Information Retrieval, and Life Utility—reveals a negative correlation between SFT performance gains and the proportion of reasoning-demanding samples. SFT models tend to memorize scoring patterns for simple tasks but fail when required to verify evidence or identify subtle factual errors. This suggests that judgment is not merely a scoring exercise but an inherently reasoning-intensive task that SFT fails to capture fully.
Methodology: The JudgeLRM Framework
To address the SFT ceiling, we propose JudgeLRM, a family of models (3B to 14B parameters) that treats evaluation as a reasoning problem solved through Reinforcement Learning (RL).
Training and Reward Design
Using Qwen2.5 and Qwen3 as base models, we employ Group Relative Policy Optimization (GRPO). The core innovation is a multi-component, outcome-driven reward function:
- Structural Reward: Ensures adherence to
<think>...</think>and<answer>...</answer>formatting. - Relation Reward: The primary component, rewarding correct identification of the superior assistant in pairwise comparisons.
- Absolute Reward: Aligns numerical scores (1-10) with ground truth.
- Confidence Reward: Encourages decisive judgments on difficult samples.
This RL process forces the model to explore diverse reasoning paths within its "thought" block, leading to emergent behaviors such as self-reflection and claim verification.
Experimental Results
JudgeLRM was evaluated using the JudgeLM dataset (GPT-4 grounded) and the PandaLM dataset (human grounded). The results demonstrate that JudgeLRM significantly outperforms SFT baselines and proprietary models.
Key Findings
- Superiority over GPT-4 and DeepSeek-R1: JudgeLRM-3B/4B exceeds GPT-4 on the human-grounded PandaLM benchmark. Larger variants (7B/8B/14B) outperform DeepSeek-R1 by over 2% in F1 score.
- Reasoning-Heavy Performance: The most substantial gains were observed in domains with high reasoning requirements, where RL successfully activated capabilities that SFT could not reach.
- Bias Mitigation: JudgeLRM shows significantly lower positional bias compared to base models and SFT-trained counterparts.
Conclusion
JudgeLRM demonstrates that Reinforcement Learning is essential for developing high-quality LLM judges. By moving beyond pattern mimicry to outcome-driven reasoning, JudgeLRM provides a robust, open-source alternative for complex evaluation tasks. Future research will focus on scaling these models and extending their capabilities to specialized expert domains such as law and medicine.
Summary updated