InstaJudge: Aligning Judgment Bias of LLM-as-Judge with Humans in Industry Applications

Myeongjun Erik Jang, Fran SilavongEMNLP 2025

TL;DR

This paper introduces INSTAJUDGE, a Python library designed to align the judgment biases of Large Language Models (LLMs) acting as judges with human evaluators in industrial settings. It achieves this through automatic prompt optimization (APO) and a novel few-shot sampling method (DPFS), significantly improving human alignment and cost-efficiency compared to existing methods.

What Problem Does This Paper Address?

Evaluating the quality of AI applications, especially those powered by LLMs, is crucial for industrial deployment. While LLM-as-Judge offers a cost-effective alternative to human evaluation, a significant challenge is the misalignment of judgment biases between humans and LLMs. Traditional solutions like preference fine-tuning are often impractical for industry due to the scarcity of domain-specific data and the inability to fine-tune closed-source LLM APIs. This paper addresses the need for a practical and effective method to align LLM judgments with human preferences in real-world business applications.

Methods

The core of INSTAJUDGE is its Automatic Prompt Optimization (APO) framework. Instead of fine-tuning, INSTAJUDGE leverages APO to discover optimal task instructions and few-shot demonstrations that maximize human alignment. The library integrates existing APO methods like DSPy (which uses Bayesian optimization for efficient grid search) and AdalFlow (which uses TextGrad for iterative prompt refinement).

A key novelty is the Distribution Preserving Few-shot Sampling (DPFS) method. DPFS constructs few-shot examples by preserving the label distribution of the training data, aiming to better align LLM evaluation criteria with human annotators. This method uses only human-written data and can be used independently or with other APO methods.

Additionally, INSTAJUDGE incorporates self-consistency decoding to enhance the robustness and precision of LLM predictions by aggregating multiple reasoning paths.

Datasets

The paper uses several industry-specific datasets for evaluation:

  • Customer Chatbot Conversation (CCC): Conversation logs from a customer banking chatbot.
  • Customer Issue: Brief summaries of customer issues.
  • SearchQA: User questions, their responses, and retrieved citations.
  • JudgeBench: A publicly available dataset for evaluating LLM-based judges, where the task is to select the better response from two LLM-generated candidates.

Key Insights and Ideas

  • APO is a viable alternative to fine-tuning: Prompt engineering, specifically through APO, can effectively inject desired judgment biases into LLMs, making it a practical approach for industrial applications.
  • DPFS is crucial for human alignment: Preserving the natural distribution of human judgments in few-shot examples significantly improves LLM alignment with human evaluators.
  • Cost-efficiency matters: DPFS offers substantial improvements in alignment while requiring significantly fewer LLM calls compared to other APO methods, making it more cost-effective.
  • Task complexity influences performance: Longer input texts (as seen in SearchQA-RQ) pose a greater challenge for LLMs, potentially degrading reasoning ability and impacting alignment.
  • DSPy and AdalFlow have complementary strengths: DSPy, with its Bayesian optimization, excels in complex tasks, while AdalFlow, using TextGrad, is advantageous for simpler tasks where iterative feedback is effective.

Key Results

  • INSTAJUDGE with DPFS achieved significant improvements in human alignment across various industrial use-cases, outperforming existing libraries like DeepEval by a large margin.
  • DPFS alone showed a 56% average improvement in alignment compared to zero-shot LLM-as-Judges.
  • The combined approach, Inst_DPFS, achieved the greatest performance gain, with an average increase of 64% compared to zero-shot Judge.
  • APO methods generally improved human alignment, with Inst_DPFS and DPFS showing the most substantial gains.
  • Self-consistency decoding provided an average performance improvement of 6%.
  • DPFS was found to be 2.2 to 5.1 times faster than DSPy and AdalFlow on average.

Limitations and Future Work

  • Challenging tasks with lengthy inputs: While INSTAJUDGE shows improvements, modest enhancements were observed in tasks involving lengthy text inputs. Future work should focus on developing more advanced approaches for these scenarios.
  • Resource intensity of self-consistency: While effective, self-consistency decoding increases LLM calls. Future work could explore more resource-efficient decoding strategies, such as adaptive self-consistency.
  • Randomness in DPFS: When initial prompts perform poorly, DPFS's selection of candidate sets based on LLM errors can lead to increased randomness. Future work aims to minimize this randomness for more consistent performance.
  • Sufficient training data: Experiments on the JudgeBench dataset suggest that a sufficient amount of training data is crucial for effective APO.

Summary updated

BibTeX
@inproceedings{jang2025instajudge,
  author = {Myeongjun Erik Jang and Fran Silavong},
  title = {{InstaJudge: Aligning Judgment Bias of LLM-as-Judge with Humans in Industry Applications}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure