JudgeLM: Fine-tuned Large Language Models are Scalable Judges

Lianghui Zhu, Xinggang Wang, Xinlong Wang

Abstract

Evaluating Large Language Models (LLMs) in open-ended scenarios remains a significant challenge. While human evaluation is the gold standard, its cost and lack of scalability have led researchers toward "LLM-as-a-judge" approaches, typically relying on proprietary models like GPT-4. However, these closed-source judges introduce issues regarding cost, privacy, and reproducibility. This paper introduces JudgeLM, a framework for fine-tuning open-source LLMs to act as high-quality, scalable judges. By leveraging a massive 100K-sample dataset and novel bias-mitigation techniques, JudgeLM achieves over 90% agreement with GPT-4, surpassing typical human-to-human agreement levels while remaining significantly faster and more cost-effective.

Introduction

As LLMs advance in creative writing and complex reasoning, traditional metrics such as BLEU or ROUGE have become inadequate. While GPT-4 has emerged as a popular surrogate for human judgment, it suffers from high costs, data privacy risks, and volatility due to model updates. Furthermore, LLM judges are prone to inherent biases, including position bias (preferring the first response), knowledge bias (relying on internal hallucinations), and format bias. JudgeLM addresses these limitations by providing a robust, open-source alternative fine-tuned specifically for the judging task.

Methods and Dataset

The JudgeLM framework fine-tunes the Vicuna series (7B, 13B, and 33B) using a "grading, judging, and reasoning" pattern. Central to this process is the JudgeLM-100K dataset, which comprises 105K seed tasks from sources like Alpaca and ShareGPT, paired with answers from 11 different LLMs and teacher judgments from GPT-4.

To ensure scalability and robustness, three key techniques were implemented:

  1. Swap Augmentation: To mitigate position bias, the model is trained on answer pairs in both possible orders (A-B and B-A).
  2. Reference Support: Including "gold standard" reference answers during training helps the judge evaluate tasks beyond its own pre-trained knowledge.
  3. Reference Drop: Randomly omitting reference answers during training ensures the model can still function in real-world scenarios where ground-truth answers are unavailable.
The JudgeLM training pipeline incorporating swap augmentation and reference support/drop techniques to produce scalable open-source judges.

Key Insights and Results

Analysis of JudgeLM reveals several critical findings regarding the nature of LLM-based evaluation:

  • Scaling Laws for Judges: Performance improves consistently with both model size and training data volume. JudgeLM-33B achieved a 90.7% agreement rate with GPT-4, notably higher than the ~82% agreement typically found between human judges on benchmarks like MT-bench.
  • Bias Mitigation: Without swap augmentation, models frequently exhibit strong position bias. With it, JudgeLM-33B reached a consistency score of over 91%.
  • Efficiency: JudgeLM-7B can evaluate 5,000 response pairs in approximately 3 minutes using 8 A100 GPUs, representing a 133x speedup over non-parallelized baselines.

Generalization and Versatility

Despite being trained primarily on text pairs, JudgeLM demonstrates remarkable zero-shot generalization across diverse evaluation formats.

JudgeLM generalizes across various evaluation scenarios, including single-answer grading, multi-turn chat, and multimodal tasks.

The model effectively handles multimodal tasks (e.g., judging image-based QA), multi-turn conversations, and absolute grading of single answers. It also shows robustness in out-of-distribution scenarios, such as identifying harmful content in the ToxicChat benchmark.

Conclusion

JudgeLM provides a high-performance, open-source solution for the scalable evaluation of LLMs. By effectively addressing common biases and achieving high alignment with proprietary models, it offers a sustainable path forward for LLM development and benchmarking. Future work will focus on expanding query diversity and applying the framework to even more capable base models.

Summary updated

BibTeX
@inproceedings{zhu2025judgelm,
  author = {Lianghui Zhu and Xinggang Wang and Xinlong Wang},
  title = {{JudgeLM: Fine-tuned Large Language Models are Scalable Judges}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=xsELpEPn4A},
}
Figure