Graders Should Cheat: Privileged Information Enables Expert-Level Automated Evaluations

Jin Peng Zhou, Séb Arnold, Nan Ding, Kilian Q Weinberger, Nan Hua, Fei Sha

Sure, here's a blog-style summary of the paper "Graders Should Cheat: Privileged Information Enables Expert-Level Automated Evaluations":

TL;DR

This paper introduces "privileged information" (PI) – extra context given to AI graders but not to the models being evaluated – to significantly improve automated evaluations, especially for challenging "frontier" tasks. By equipping AI graders with PI like ground-truth solutions or detailed guidelines, they can achieve expert-level performance, even surpassing human evaluators in some cases, and enable more nuanced model comparisons.

What Problem Does This Paper Address?

Evaluating the performance of large language models (LLMs) is crucial for tracking progress and developing better AI. However, traditional automated evaluation methods often struggle with "frontier" tasks – those that are at the edge of current LLM capabilities, like Olympiad-level math or graduate-level STEM problems. In these domains, LLMs themselves are not proficient enough to reliably evaluate each other, leading to noisy and unreliable assessments. This creates a paradox: how can a weaker model accurately grade a stronger one on tasks that are difficult for both? The paper aims to bridge this gap by enhancing the capabilities of AI graders.

Methods

The core innovation is the use of privileged information (PI). PI is any auxiliary context provided to the grader LLM that is not available to the candidate LLMs being evaluated. The authors explore several types of PI:

  • Ground-truth solutions: The correct answer or a step-by-step derivation for a problem.
  • Rating guidelines: Prompt-specific instructions on what to look for in a good response.
  • Prior ratings: Synthesized or human-provided examples of previous evaluations with rationales.
  • Multimodal annotations: Such as image captions for visual tasks.

A key technique is the tiered evaluation strategy. For difficult problems, PI (like step-by-step solutions) is used to generate a series of instructional "hints." These hints incrementally simplify the problem, allowing for a more granular assessment of model capabilities across different difficulty levels without requiring new annotations. This approach is model-agnostic and doesn't require retraining the grader.

Datasets

The paper evaluates its approach on several benchmarks:

  • RewardBench: A benchmark for evaluating LLMs' ability to model human preferences, with categories like Chat, Chat Hard, Safety, and Reasoning.
  • Vibe-Eval: A visual question-answering benchmark with normal and hard splits.
  • MathOdyssey (Olympiad subset): A challenging benchmark of Olympiad-level math problems.
  • MATH-Adv and GPQA: Adversarially selected subsets of challenging reasoning datasets.

Key Insights and Ideas

  • "Graders should cheat": The title's provocative claim highlights that giving AI graders an "unfair advantage" (PI) is essential for them to perform well on difficult tasks.
  • PI bridges the capability gap: PI allows weaker graders to effectively evaluate stronger candidate models on tasks that are beyond their independent capabilities.
  • Tiered evaluation for nuanced insights: Decomposing difficult problems into hints allows for a more precise understanding of how models perform at different difficulty levels, revealing subtle differences in their abilities.
  • Compositionality of PI: Combining different types of PI (e.g., image captions, guidelines, and reference answers) leads to further improvements in grading performance.
  • Human graders also benefit: PI not only boosts AI grader performance but also improves human evaluator consistency and accuracy.

Key Results

  • RewardBench: Graders augmented with PI achieved state-of-the-art performance, surpassing non-PI counterparts by over 6 percentage points in accuracy.
  • Vibe-Eval: PI enabled graders to surpass individual human graders, improving correlation with human averages by over 0.35 points.
  • MathOdyssey: Graders with PI achieved a Spearman correlation of up to 0.71 with expert human ratings, approaching expert-level quality.
  • Bias Reduction: PI significantly reduced verbosity and formatting biases in LLM graders.
  • Model Separability: Tiered hints dramatically improved model separation on challenging reasoning tasks, moving performance from near-random chance to over 80% accuracy on MATH-Adv.

Limitations and Future Work

The authors acknowledge several limitations:

  • Pedagogical quality of hints: While hints are derived from ground truth and are generally accurate, their pedagogical quality hasn't been formally validated by human experts.
  • Scalability of human-authored PI: While human-authored PI provides high-quality guidance, producing it at scale for frontier benchmarks can be costly.
  • Potential for amplified modeling errors: Relying solely on automatically synthesized PI might amplify existing errors.

Future work could explore:

  • Domain-general PI: Developing reusable PI like general rating guidelines or best practices to balance cost and coverage.
  • Hybrid strategies: Combining limited expert-provided PI with broader automated synthesis.
  • Combating self-enhancement bias: Investigating techniques like response style normalization or alternative prompting strategies.
  • Model training insights: Exploring how sensitivity to hints during training might improve model capabilities.

Summary updated

Abstract

Auto-evaluating language models (LMs), i.e., using a grader LM to evaluate the candidate LM, is an appealing way to accelerate the evaluation process and the cost associated with it. But this presents a paradox: how can we trust the grader LM, which is presumably weaker than the candidate LM, to assess problems that are beyond the frontier of the capabilities of either model or both? For instance, today's LMs struggle on graduate-level physics and Olympiad-level math, making them unreliable graders in these domains. We show that providing privileged information -- such as ground-truth solutions or problem-specific guidelines -- improves automated evaluations on such frontier problems. This approach offers two key advantages. First, it expands the range of problems where LMs graders apply. Specifically, weaker models can now rate the predictions of stronger models. Second, privileged information can be used to devise easier variations of challenging problems which improves the separability of different LMs on tasks where their performance is generally low. With this approach, general-purpose LM graders match the state of the art performance on RewardBench, surpassing almost all the specially-tuned models. LM graders also outperform individual human raters on Vibe-Eval, and approach human expert graders on Olympiad-level math problems.
BibTeX
@inproceedings{zhou2025graders,
  author = {Jin Peng Zhou and Séb Arnold and Nan Ding and Kilian Q Weinberger and Nan Hua and Fei Sha},
  title = {{Graders Should Cheat: Privileged Information Enables Expert-Level Automated Evaluations}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure