Large Language Models (LLMs) have revolutionized how we evaluate complex, open-ended generative tasks. From grading student essays to judging image captions, frameworks like CLAIR and LLMScore have shown that an LLM prompt can replace complex, hand-engineered metrics while achieving high correlation with human judgment.
However, treating an LLM as a traditional evaluation metric introduces a fundamental flaw: traditional metrics are deterministic mathematical functions, whereas LLMs are probabilistic text generators. When an LLM acts as a judge, its stochastic nature means that evaluating the exact same input multiple times can yield entirely different scores. This phenomenon, known as stochastic evaluation inconsistency, threatens the reliability of automated benchmarking, model alignment, and grading systems.
The Mechanics of Evaluator Instability
Why do LLM judges disagree with themselves? At its core, the problem stems from the decoding process. When an LLM generates an evaluation score, it samples tokens sequentially from a probability distribution. Even when the temperature is set to zero to encourage greedy decoding, subtle variations in context, underlying hardware non-determinism, or complex prompt structures can cause the model to diverge down different reasoning paths.
As Lee et al. demonstrate in Evaluating the Consistency of LLM Evaluators, this instability manifests in two distinct ways:
- Low Self-Consistency: Assigning different scores to the identical input across multiple independent inference trials.
- Low Inter-Scale Consistency: Failing to maintain a stable metric when the scoring format changes (e.g., switching from a 1-5 scale to a 1-100 scale).
Furthermore, assigning absolute numerical scores is inherently difficult for LLMs because they lack a stable internal rubric. As noted by Bi et al. in LINKAGE, this lack of a fixed scale causes the model's absolute ratings to fluctuate wildly across different evaluation sessions. In complex domains like video captioning, Chen et al. found that some automated benchmarks achieve consistent scoring in less than half of repeated trials, highlighting how severe this noise can become (VidCapBench).
The Real-World Consequences of Noisy Metrics
The stochasticity of LLM judges is not just a theoretical nuisance; it introduces severe vulnerabilities in practical applications.
Unreliable Leaderboards
In The Good, The Bad, and The Greedy, Song et al. point out that most public leaderboards rely on a single evaluation run. This ignores the wide distribution of potential outcomes, meaning a model's high rank might simply be a lucky draw from a stochastic distribution rather than a reflection of genuine capability.
Exploitation and Prompt-Hacking
When deployed in high-stakes scenarios like education, stochasticity undermines fairness. Chiang et al. studied the real-world deployment of an LLM grader in a large course and observed a "cat-and-mouse" game where students repeatedly resubmitted the exact same assignment to exploit the model's non-determinism until a higher grade was randomly generated (Large Language Model as an Assignment Evaluator).
Unstable Model Alignment
During Reinforcement Learning from Human Feedback (RLHF), generative reward models may sample rubrics dynamically. As argued by Gupta et al. in Unveiling Context-Aware Criteria in Self-Assessing LLMs, this probabilistic bias yields inconsistent criteria across runs, destabilizing the reward signal and leading to reward hacking.
Strategies for Taming Stochastic Variance
To transform LLMs into reliable evaluation tools, researchers have developed several architectural and statistical mitigation strategies.
Statistical Ensembling and Expected Values
The most straightforward way to combat stochastic noise is to move away from single-pass generation. In Evaluating Evaluation Metrics, Xiao et al. propose a framework based on measurement theory, demonstrating that averaging scores over multiple iterations effectively dampens non-deterministic noise.
Taking this a step further, Tong et al. design G-VEval to calculate an explicit "expected value" derived directly from the model's token probability distribution, bypassing the variance of free-form text generation entirely. For long-form texts where context shifts exacerbate instability, Yang et al. show that multi-pass score averaging is essential to achieve stable assessments (What Matters in Evaluating Book-Length Stories?).
Multi-Agent Discussion Frameworks
Rather than relying on a single isolated judge, some approaches introduce collaborative deliberation to filter out individual stochastic errors. In MATEval, Li et al. introduce a multi-agent discussion framework where multiple LLM instances debate the merits of a piece of text. This collaborative process forces the agents to provide transparent rationales, catching subtle hallucinations or logical inconsistencies that a single noisy agent would overlook.
Latent and Reference-Based Calibration
Another paradigm shifts the evaluation from open-ended text generation to structured or internal representations. Girrbach et al. introduce "Latent Judges" in Reference-Free Rating of LLM Responses via Latent Information, a method that extracts deterministic quality scores directly from a model's internal probabilities and hidden activations. By avoiding stochastic decoding entirely, this approach achieves high self-agreement and finer granularity than traditional text-based ratings.
[Input Text] ───► [LLM Forward Pass] ───► [Hidden Activations & Token Probabilities] ───► [Deterministic Score]
(Bypasses Stochastic Decoding)
Alternatively, frameworks like LINKAGE anchor the model's judgment by forcing it to perform listwise ranking against a set of pre-graded reference answers, providing an explicit external "ruler" to stabilize the evaluation scale.
Key Foundational Work
Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theory (2023) — contributed 2 concept elements to this direction.
CLAIR: Evaluating Image Captions with Large Language Models (2023) — contributed 1 concept elements to this direction.
Arithmetic Sampling: Parallel Diverse Decoding for Large Language Models (2023) — contributed 1 concept elements to this direction.