Abstract
The evaluation paradigm of LLM-as-judge gains popularity due to its significant reduction in human labor and time costs. This approach utilizes one or more large language models (LLMs) to assess the quality of outputs from other LLMs. However, existing methods rely on generic scoring rubrics that fail to consider the specificities of each question and its problem-solving process, compromising precision and stability in assessments. Inspired by human examination scoring processes, we propose a new evaluation paradigm based on self-adaptive rubrics. Specifically, we create detailed scoring rubrics for each question, capturing the primary and secondary criteria in a structured format of scoring and deduction points that mimic a human evaluator's analytical process. Building on this paradigm, we further develop a novel benchmark called SedarEval, which covers a range of domains including long-tail knowledge, mathematics, coding, and logical reasoning. SedarEval consists of 1,000 meticulously crafted questions, each with its own self-adaptive rubric. To further streamline the evaluation, we train a specialized evaluator language model (evaluator LM) to supplant human graders. Using the same training data, our evaluator LM achieves a higher concordance rate with human grading results than other paradigms, including GPT-4, highlighting the superiority and efficiency of our approach.
Introduction
The "LLM-as-judge" paradigm has become the standard for evaluating open-ended model responses, yet it remains hindered by the use of generic rubrics. Current benchmarks often rely on broad principles—such as being "helpful" or "honest"—to score every response regardless of the task type. This "one-size-fits-all" approach fails to capture the nuances required for specialized domains like mathematics or coding, often leading to inconsistent scores and "hallucinated" evaluations where the judge model relies on its own internal knowledge rather than objective criteria.
SedarEval addresses this by introducing self-adaptive rubrics. By shifting the evaluation task from subjective judgment to strict instruction following, we provide automated judges with a precise rulebook that mimics the analytical process of human graders.
The Self-Adaptive Rubric Paradigm
The core innovation of SedarEval is the structured, question-specific rubric. Instead of a single scoring guide, every question in the benchmark is paired with its own rubric containing:
- Scoring Points: Specific positive increments for correct elements (e.g., identifying a specific historical figure).
- Penalty Points: Specific deductions for errors (e.g., factual mistakes regarding dates).
- Background Knowledge: Contextual information provided to the judge to ensure it does not need to rely on potentially flawed internal knowledge to determine the ground truth.
To scale this approach, we developed specialized evaluator models (the "XD" series) trained to follow these granular instructions. We also utilized a Human-AI consistency strategy, filtering training data to retain only examples where model evaluations aligned perfectly with human scores.
The SedarEval Benchmark
SedarEval consists of 1,000 high-quality questions across eight major categories, including logic, math, coding, and long-tail knowledge. To ensure the benchmark's robustness, we collected responses from 20 different LLMs—including GPT-4, Claude Opus, and various Chinese-proficient models—to serve as the evaluation targets.
Our experiments revealed a "scaling law" for benchmarks: model rankings tend to stabilize once a benchmark reaches approximately 300 questions. This provides a useful heuristic for researchers designing future evaluation sets.
Key Results and Performance
The SedarEval approach significantly outperforms traditional evaluation paradigms. Our specialized evaluator model, XD-v4, achieved a higher concordance rate (Accuracy) with human graders than GPT-4.
As shown in the comparative analysis, the SedarEval paradigm (represented by the XD model) demonstrates superior alignment with human judgment at the question level compared to GPT-3.5. While GPT-4 remains a strong baseline, our optimized XD-v4 model—trained specifically on self-adaptive rubrics—surpasses it in human alignment accuracy (0.664 vs 0.654). Furthermore, the approach proved robust against score distribution shifts, maintaining high correlation with human ground truth even when test data distributions differed from training data.
Conclusion
SedarEval demonstrates that the key to reliable automated evaluation lies in the granularity of the rubric. By treating evaluation as an instruction-following task, we reduce the cognitive load on judge models and achieve human-level precision. Future work will focus on automating the rubric generation process and extending the framework to handle multi-solution problems where several distinct answers may be equally valid.
Summary updated