J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning

Chenxi Whitehouse, Tianlu Wang, Ping Yu, Xian Li, Jason Weston, Ilia Kulikov, Swarnadeep Saha

TL;DR

This paper introduces J1, a novel reinforcement learning framework that trains Large Language Models (LLMs) to "think before they judge." By unifying diverse evaluation tasks into a single format optimized with verifiable rewards, J1 significantly improves the quality and consistency of LLM-based evaluations, achieving state-of-the-art performance across multiple benchmarks.

What Problem Does This Paper Address?

The rapid advancement of AI models, particularly LLMs, has created a bottleneck in evaluation. Traditional methods often rely on simple scoring, failing to capture the nuanced reasoning required for robust assessment. LLM-as-a-Judge approaches offer a promising solution by enabling chain-of-thought reasoning, but optimizing this reasoning process itself is a significant challenge. Existing methods struggle with issues like positional bias (where the order of responses affects the judgment) and a lack of generalizability across different types of prompts (verifiable, like math problems, and non-verifiable, like creative writing). This paper aims to address these limitations by developing a method to train LLM judges that can effectively reason, generalize, and provide consistent, high-quality evaluations.

Methods

The core innovation of J1 is its reinforcement learning (RL) framework designed to incentivize "thinking" in LLM judges. Key methodological components include:

  • Unified Verifiable Training: J1 converts all judgment tasks, regardless of whether they are verifiable (e.g., math) or non-verifiable (e.g., user prompts), into a unified format. This allows for optimization using RL with verifiable rewards, mitigating positional bias and enabling training of a single, generalist judge.
  • Reasoning-Optimized Training: The framework uses GRPO (a specific RL algorithm) to directly optimize the quality of the evaluation "thoughts" (chain-of-thought reasoning). This is guided by seed prompts and targeted reward schemes, teaching the model to reason critically.
  • Multitask and Bias-Aware Judge: J1 addresses positional bias through consistency-based rewards. It also introduces a novel approach to train inherently consistent pointwise judges using only pairwise supervision. These are then unified into a single multitask model capable of both pointwise and pairwise evaluations.

Datasets

  • Training Data: J1 is trained on a synthetic dataset of 22,000 preference pairs, comprising 17,000 from WildChat (non-verifiable prompts) and 5,000 from MATH (verifiable prompts). This synthetic data generation process is crucial for enabling RL training with verifiable rewards.
  • Evaluation Benchmarks: J1 is evaluated on several established benchmarks, including:
  • PPE (Preference Proxy Evaluations): A large-scale benchmark linking reward models to human preferences, with subsets for both preference and correctness on verifiable tasks.
  • JudgeBench: Contains challenging response pairs across knowledge, reasoning, math, and coding.
  • RM-Bench: Assesses reward model robustness to content differences and style biases.
  • FollowBenchEval: Tests reward models' ability to validate multi-level constraints.
  • RewardBench: A comprehensive benchmark for generative reward models.

Key Insights and Ideas

  • Thinking is Key: The paper strongly emphasizes that optimizing the "thinking" process of LLM judges, not just their final output, is critical for improving evaluation quality.
  • Unified Training for Generalization: By converting diverse tasks into a single, verifiable format, J1 demonstrates that a single model can effectively judge a wide range of prompts, a significant step towards generalist AI evaluators.
  • Synthetic Data is Sufficient: J1 achieves state-of-the-art results using only synthetic training data, reducing reliance on expensive human annotations.
  • Pointwise Judges Can Be Consistent: The work shows that pointwise judges, which evaluate responses individually, can be trained to be position-consistent, addressing a key limitation of traditional pairwise judges.
  • Multitasking Boosts Performance: Combining pointwise and pairwise evaluation capabilities into a single multitask model leads to superior performance compared to separately trained models.

Key Results

  • State-of-the-Art Performance: J1 models, particularly J1-Qwen-32B-MultiTask, achieve state-of-the-art results across multiple benchmarks, often by significant margins. For instance, on PPE Correctness, J1-Qwen-32B-MultiTask reached 76.8% accuracy, outperforming previous methods.
  • Outperforming Larger Models: J1-Qwen-32B (a 32B model) outperforms much larger models like DeepSeek-R1-671B and established judges like o1-mini and o3 on several benchmarks, showcasing the efficiency of the J1 approach.
  • Improved Position Consistency: J1 models demonstrate improved position-consistent accuracy and reduced verdict flips, indicating better handling of positional bias.
  • Qualitative Improvements: J1 models learn to generate systematic evaluation strategies, including dynamic criteria generation, creating reference answers, iterative self-correction, and providing detailed feedback.

Limitations and Future Work

  • Reliance on Synthetic Data: While effective, the reliance on synthetic data might not perfectly capture all nuances of real-world human preferences. Future work could explore incorporating more diverse or human-annotated data.
  • Test-Time Scaling: The paper shows that test-time scaling (e.g., averaging scores over multiple generations) can further improve performance, suggesting that exploring more sophisticated test-time strategies could be beneficial.
  • Further Exploration of Reward Schemes: While the paper ablates some reward components, further research into designing even more effective reward signals for complex reasoning tasks could be explored.
  • Generalizability to Novel Tasks: While J1 shows strong generalization, its performance on entirely novel or highly specialized evaluation tasks could be further investigated.

Summary updated

Abstract

The progress of AI is bottlenecked by the quality of evaluation, making powerful LLM-as-a-Judge models a core solution. The efficacy of these judges depends on their chain-of-thought reasoning, creating a critical need for methods that can effectively optimize this reasoning process. In this work, we introduce J1, a reinforcement learning framework for teaching LLM judges to think before making decisions. Our core contribution lies in converting all judgment tasks for non-verifiable and verifiable prompts into a unified format with verifiable rewards, enabling direct optimization of evaluation quality while mitigating positional bias. We then use RL to train thinking-judges at scales of 8B, 32B, and 70B and show that they obtain state-of-the-art performance across multiple benchmarks. In particular, J1-Qwen-32B, our multitasked pointwise and pairwise judge also outperforms o1-mini, o3, and a much larger 671B DeepSeek-R1 on some benchmarks, while only training on synthetic data. Through comprehensive ablations of pairwise, pointwise, and multitask J1 variants, we demonstrate the effectiveness of our approach across seed prompts, reward strategies, and training recipes. Qualitative analysis reveals that J1 develops systematic evaluation strategies, including dynamic criteria generation, reference answer creation, iterative self-correction of initial assessments, and feedback generation for low-quality responses.
BibTeX
@inproceedings{whitehouse2026j,
  author = {Chenxi Whitehouse and Tianlu Wang and Ping Yu and Xian Li and Jason Weston and Ilia Kulikov and Swarnadeep Saha},
  title = {{J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=dnJEHl6DI1},
}
Figure