Doubly-Robust LLM-as-a-Judge: Externally Valid Estimation with Imperfect Personas

Luke Guerdan, Justin Whitehouse, Kimberly Le Truong, Kenneth Holstein, Zhiwei Steven Wu

Introduction

Evaluating Generative AI (GenAI) systems often relies on estimating quality metrics like helpfulness or safety. However, these estimates frequently suffer from evaluation sampling bias, which compromises their external validity. This bias typically arises from two sources:

  1. Covariate Shift: The demographic profile of lab raters (e.g., crowdworkers) differs from the actual target user population.
  2. Selection Bias: Raters may disproportionately opt-out or fail quality checks when encountering specific types of sensitive or complex content.

Existing methods like Prediction Powered Inference (PPI++) often assume data is "missing completely at random," a condition rarely met in practice. This paper introduces a doubly-robust framework to produce statistically valid estimates even when human data is skewed and LLM judges are imperfect proxies.

The proposed framework leverages LLM-as-a-Judge with persona prompting to bridge the gap between biased source distributions and target deployment distributions.

Methods: The Doubly-Robust Framework

The authors propose a Doubly-Robust (DR) Estimator that remains valid if either an Outcome Model (predicting human ratings from LLM personas) or a Reweighting Model (correcting for sampling bias) is correctly specified.

Direct Ratio Estimation via Riesz Loss

Traditional reweighting struggles in high-dimensional spaces like text embeddings. To address this, the framework employs a "Riesz loss" approach. Instead of estimating source and target densities separately—which can compound errors—it directly learns the density ratio that minimizes a specific loss function. This provides greater stability when handling complex data.

LLM Personas as Informative Signals

A key insight is that LLM "personas" do not need to be perfect replicas of human behavior. As long as there is a correlation between the LLM's simulated rating and the actual human's rating, the DR framework can utilize this signal to reduce variance and improve estimation accuracy.

Key Results

The effectiveness of the DR (Riesz) estimator was evaluated across three settings: a fully synthetic environment and two semi-synthetic datasets (PRISM for helpfulness and DICES for harmfulness).

Comparison of the DR (Riesz) estimator against baselines. The DR (Riesz) method consistently aligns with the target mean (black dashed line) while maintaining valid confidence intervals, unlike simpler averaging or persona-only methods.

The results highlight several critical findings:

  • Valid Coverage: In high-bias scenarios, DR (Riesz) maintained a ~95% coverage rate, whereas other methods often missed the true value significantly.
  • Reduced Bias: The method achieved lower Mean Absolute Error (MAE) compared to simple sample averaging or persona-only baselines.
  • Robustness: The framework remained reliable even when using "weaker" LLM judges, demonstrating that the synergy of biased human samples and imperfect LLM simulations can yield statistically sound results.

Conclusion

This work provides a rigorous path forward for AI evaluation by acknowledging and correcting for the inherent biases in human rating processes. By combining imperfect LLM judges with biased human data through a doubly-robust lens, practitioners can obtain more accurate predictions of how models will perform in real-world deployment. Future research may look into relaxing the "no concept drift" assumption and applying these techniques to non-text modalities.

Summary updated

BibTeX
@inproceedings{guerdan2026doubly,
  author = {Luke Guerdan and Justin Whitehouse and Kimberly Le Truong and Kenneth Holstein and Zhiwei Steven Wu},
  title = {{Doubly-Robust LLM-as-a-Judge: Externally Valid Estimation with Imperfect Personas}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=EIA1tpKYL7},
}
Figure