Skewed Score: A statistical framework to assess autograders

Magda Dubois, Harry Coppock, Mario Giulianelli, Timo Flesch, Lennart Luettgau, Cozmin Ududec

Abstract

Evaluation of Large Language Models (LLMs) increasingly relies on "autograders"—LLMs used to score the outputs of other models. However, these automated judges often harbor systematic biases, such as preferences for longer responses or models from the same family. We propose a unified statistical framework based on Bayesian Generalized Linear Models (GLMs) that moves beyond simple correlation metrics. This approach allows for the simultaneous estimation of model performance and grader bias while providing rigorous uncertainty estimates through credible intervals.

Introduction

As LLMs tackle more nuanced tasks, human evaluation has become a significant bottleneck. To scale, researchers have turned to "LLM-as-a-judge" systems. While efficient, these autograders are not neutral; they exhibit "length bias" and "self-bias." Traditional reliability checks, such as Pearson correlation or Krippendorff’s alpha, provide a "black box" view of disagreement, failing to distinguish between random noise and systematic grader harshness or favoritism.

Methodology: Bayesian GLMs

The proposed framework utilizes Bayesian GLMs to model evaluation outcomes as a function of multiple predictors: the grader (human vs. specific LLM), the item (the model being graded), and their interactions. Key technical components include:

  • Ordered Logistic Regression: Used for absolute scores (e.g., 1–10) to treat the scale as ordinal, acknowledging that the perceived distance between scores (e.g., 5 to 6 vs. 9 to 10) may vary.
  • Binomial Likelihood: Employed for pairwise comparisons (Model A vs. Model B).
  • Hierarchical Modeling: Enables "partial pooling," allowing the model to learn group-level characteristics (e.g., the average behavior of all human graders) while still accounting for individual rater variance.
  • Bayesian Inference: Provides full posterior distributions rather than point estimates, offering credible intervals that are essential for interpreting the certainty of detected biases.

Simultaneous Evaluation and Bias Detection

A primary advantage of this framework is the ability to evaluate the LLM and the grader concurrently. By including interaction terms, researchers can pinpoint specific biases, such as an autograder favoring a specific model family.

Score distributions and posterior parameter estimates for multiple graders and LLMs, illustrating the detection of systematic biases and interaction effects.

As shown in Figure 4, the framework can isolate interaction effects (e.g., Grader × LLM [Autograder A, LLM A]), revealing when a grader is systematically more lenient toward a specific model. This allows for "bias-corrected agreement" calculations—determining what the agreement would look like if systematic shifts in grader behavior were removed.

Hierarchical Modeling and Uncertainty

In real-world scenarios with multiple raters, the framework's hierarchical nature becomes invaluable. It can aggregate data across many humans and autograders to find a "consensus" while identifying outliers.

Posterior estimates for a larger cohort of human and automated graders, demonstrating hierarchical pooling and individual rater variability.

Figure 6 demonstrates how the model estimates the "Human mean" and "Autograder mean" separately. Furthermore, the Bayesian approach highlights that in low-data regimes, point estimates can be misleading; credible intervals provide a more honest assessment of the evidence. This is particularly useful for scale calibration, helping researchers see if their rubric is working as intended across the entire score range.

Conclusion

The Bayesian GLM framework, supported by the HiBayes package, offers a principled way to audit autograders. While it requires more statistical rigor than simple correlations, it provides the transparency needed to trust automated evaluation. Future work will focus on applying this framework to large-scale, real-world datasets like Chatbot Arena to further explore how these biases manifest in diverse evaluation contexts.

Summary updated

Abstract

The evaluation of large language model (LLM) outputs is increasingly performed by other LLMs, a setup commonly known as "LLM-as-a-judge", or autograders. While autograders offer a scalable alternative to human evaluation, they have shown mixed reliability and may exhibit systematic biases, depending on response type, scoring methodology, domain specificity, or other factors. Here we propose a statistical framework based on Bayesian generalised linear models (GLMs) that enables researchers to simultaneously assess their autograders while addressing their primary research questions (e.g., LLM evaluation). Our approach models evaluation outcomes (e.g., scores or pairwise preferences) as a function of properties of the grader (e.g., human vs. autograder) and the evaluated item (e.g., response length or the LLM that generated it), allowing for explicit quantification of scoring differences and potential biases within a unified framework. In addition, our method can be used to augment traditional metrics such as inter-rater agreement, by providing uncertainty estimates and clarifying sources of disagreement. Overall, this approach contributes to more robust and interpretable use of autograders in LLM evaluation, enabling both performance analysis and bias detection.
BibTeX
@misc{dubois2025skewed,
  author = {Magda Dubois and Harry Coppock and Mario Giulianelli and Timo Flesch and Lennart Luettgau and Cozmin Ududec},
  title = {{Skewed Score: A statistical framework to assess autograders}},
  year = {2025},
  eprint = {2507.03772},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.03772},
}
Figure