When a large language model grades another model's answer, we usually check the grader against humans who were paid to read clean prompts in a controlled setting. Real chat logs contain a cheaper and more honest signal. The user's own next message often says whether the answer worked: "thanks," "that's wrong," "shorter please," the same question asked again, or nothing at all. This proposal treats that next move as an outcome measurement collected in the wild. It uses that measurement to test whether automatic graders track what real users actually wanted. The design copies an old move from experimental science: validate a cheap measurement taken under messy, uncontrolled conditions against a careful one. Here it runs in reverse. The polished automatic grader is checked against the noisy but ecologically valid behavior of a million real users.
The main difficulty is that the wild signal is censored in a biased way. Users who are badly served often leave without saying anything, so the conversations that end in silence are not a random sample. We propose a validation protocol that labels user reactions, measures how well judges agree with them, regresses the disagreement on known judge biases such as length, politeness and invented citations, and handles abandoned conversations with explicit bounds rather than ignoring them. In two sentences: an LLM judge's score for an assistant turn should predict how the real user reacted to it. Where it does not, we can measure which shortcut cues drive the gap and whether the gap survives a correction for users who quietly quit.
Background and the gap
An "LLM-as-a-judge" is a strong model prompted to score or compare other models' outputs. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena established the approach. It reports that GPT-4's pairwise verdicts agree with human experts about 80–85% of the time, roughly matching agreement between two experts, on curated two-turn MT-Bench questions and crowd-voted Arena battles. That paper motivates our work and supplies our judge protocol. It also treats explicit human votes as ground truth and never looks at long, unscripted conversations.
Two other lines of work constrain the design.
The bias audits show that judges are moved by cues unrelated to substance:
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge shows that fabricated citations and polite tone inflate scores, using GPT-4o rewrites to perturb clean benchmark answers.
- The Silent Judge shows that provenance and recency tags shift verdicts by up to 30% while judges never mention the tags in their rationales.
Both perturbations are synthetic. Neither paper asks whether these biases make judges diverge from real users on real traffic, which is the question we test.
The debiasing work fixes one bias at a time against a different target. Length-Controlled AlpacaEval fits a regression that removes the effect of answer length on the judge's own preference. It raises Spearman correlation with Chatbot Arena from 0.94 to 0.98, but it models only length, uses single-turn prompts, and calibrates the judge against itself. We reuse its regression technique with a different outcome (user reaction) and several bias covariates.
The data source is WildChat, about one million real conversations with GPT-3.5 and GPT-4 in 68 languages. The WildChat authors trained a model on these logs but evaluated it on MT-Bench. Nobody has graded the logs themselves.
The constraint that makes this nontrivial comes from outside machine learning. Validating the web-based evaluation of NLG systems found two things when comparing web and lab evaluations of the same systems:
- Silent failure. Web users who failed a task submitted feedback 16% of the time, against 93% for those who succeeded, while lab participants reported at over 94% either way.
- The payoff from scale. The larger web sample found 26 significant system differences the lab missed, with no contradictory significant results.
Both lessons carry over directly. Silence after an assistant turn is informative, and the sheer volume of wild data buys statistical power that lab studies cannot.
Core Claim
We claim three things. First, judge agreement with real user reactions on wild multi-turn conversations is measurably lower than the 80–85% lab figure. Second, a substantial part of that gap is explained by the same shortcut cues found synthetically, with length as the expected main driver: judges reward long answers that real users push back on. Third, conclusions about which judge is best depend on how abandoned conversations are handled. Simply dropping them, as current practice implicitly does, overstates agreement because it keeps only users who stayed engaged.
Research Program
Reaction labels. Each assistant turn in WildChat that is followed by a user message receives one reaction label:
- acceptance or thanks,
- correction or complaint,
- re-asking the same request,
- a style complaint such as "too long" or "just the code",
- moving to an unrelated new request.
These labels come from a classifier drawn from a different model family than any judge under test. The classifier only reads the user's follow-up and the immediate context, which is a much easier task than grading the answer. We audit it against human labels on a stratified sample of 1,500 turns, each labeled by three annotators. Turns with no follow-up are marked as censored, not as neutral.
Judge scores. Each candidate judge scores every assistant turn pointwise, given the full preceding conversation, using the rubric and chain-of-thought prompting of Judging LLM-as-a-judge. The candidates are GPT-4o, Claude-3.5-Sonnet and Qwen2-72B, the models already audited in Justice or Prejudice?.
Agreement. We collapse reactions to positive (acceptance, or a new topic after an explicit acknowledgment) versus negative (correction, re-ask, style complaint). Agreement is then measured as AUC (area under the curve) of the judge score for predicting a negative reaction, alongside pairwise agreement on matched turns.
Disagreement regression. To see what drives disagreement, we fit a logistic regression in the style of Length-Controlled AlpacaEval:
$$\Pr(\text{negative reaction}_t) = \sigma\big(\alpha\, s_t + \textstyle\sum_k \beta_k z_{tk} + \gamma^\top c_t\big)$$
The symbols are defined as follows. $s_t$ is the judge's score for turn $t$. The $z_{tk}$ are bias cues measured on the answer: length, politeness markers, citation-like strings and formatting density. $c_t$ are context controls: language, task type, turn index and backend model. $\sigma$ is the logistic function.
The reading is simple. If the judge already captures what users want, the cue coefficients $\beta_k$ should be near zero once the judge's score is included. A nonzero $\beta_k$ means the cue predicts user reactions over and above the judge. The reaction-calibrated judge we test is this model's prediction, fitted on one split and evaluated on held-out conversations.
Abandonment. Conversations ending in silence are handled by partial identification. We report each agreement metric under two extremes: all final silences counted as satisfied, and all counted as failed. We then narrow those bounds using covariates that make satisfaction plausible, such as whether the final answer completes a self-contained request. If a judge ranking flips within the bounds, that ranking is not supported by the data.
Model-level consistency. Finally, we run a Koller-style directional check at the level of whole models. Does the judge's ranking of WildChat's GPT-4 and GPT-3.5 backends, broken down by task category, agree in direction with the ranking implied by user reactions? Users partly chose which backend to use, which is itself an uncontrolled confounder. We therefore compare within matched prompt categories and languages.
Baselines
Do nothing. The naive reference is to accept the judge's lab validation and assume it transfers to wild traffic. Concretely, we predict that each judge's wild agreement equals its lab agreement on the 3,000 MT-Bench expert votes. The reported figure is about 80–85% for GPT-4 in Judging LLM-as-a-judge. We will recompute lab agreement for our judges with the same prompts. A wild agreement clearly below this figure, with confidence intervals excluding it, refutes the assumption.
Length heuristic. A judge-free baseline predicts reactions from answer length alone. If it matches the judges on wild reactions, the judges add nothing there.
Length-only correction. The strongest existing correction applies the Length-Controlled AlpacaEval regression of Length-Controlled AlpacaEval, with length as the only covariate and the judge's raw preference as the target. Its reported result is a Spearman correlation of 0.98 with Chatbot Arena, up from 0.94. There is no reported result on Lacuna for any judge's agreement with WildChat user reactions, so no wild number exists to beat. We fit the length-only and multi-cue reaction-calibrated models on identical training conversations, with the same regularization search and the same held-out split. Beating the length-only correction means two things. The multi-cue model must improve held-out AUC on user reactions. It must also not lose agreement on the human-audited sample, since gaining on users while losing on careful annotators would mean the model has merely learned user quirks.
Next-turn sentiment. A sentiment classifier applied to the user's follow-up is not a judge of the assistant turn, because it needs the follow-up. It is included to measure how much reaction signal is recoverable at all.
Abandonment ablation. The ablation that isolates the attrition ingredient compares complete-case analysis (dropping censored turns) against the bounded analysis. The ingredient matters if the judge ranking or the sign of any bias coefficient changes between the two.
Why not just…?
Why not just validate judges against Chatbot Arena votes, which are already wild? Arena votes are explicit judgments of single responses by self-selected voters who know they are evaluating. That is closer to a lab task carried out on the web than to ordinary use. The earlier validation work already reached roughly 80–85% agreement on this kind of data, and it tells us nothing about multi-turn conversations where the user's goal drifts. If wild reaction agreement turns out indistinguishable from Arena agreement, our main claim fails, and we will report that.
Why not just read the user's next message and skip the judge? In logged data, the next message is the better signal, and our sentiment baseline measures exactly that. Judges are needed where no reaction exists: offline evaluation of new models, final turns, and silent users. The question is whether a judge's offline score predicts reactions it cannot see. This is a real threat to the proposal's usefulness only if judge AUC is near chance, which would itself be a significant negative result.
Why not just drop abandoned conversations? The NLG web study shows that silence is strongly tied to outcome: 16% reporting after failure against 93% after success. Dropping silent conversations therefore keeps the users who were served well and inflates agreement. The abandonment ablation tests directly whether this changes conclusions. If the bounds are narrow and rankings are stable, complete-case analysis is acceptable, and we will say so.
Why not just pay annotators to grade WildChat turns? We do this for 1,500 turns, and these labels act as the referee. Annotators, however, read a stranger's conversation without knowing that person's intent. ObjexMT shows that even strong models extract users' hidden goals poorly in multi-turn settings, and outsiders face the same problem. Three labels per turn also let us check, following Distributional LLM-as-a-Judge, whether turns where judge and user disagree are also turns where annotators disagree. That would mark them as genuinely ambiguous rather than as judge errors.
Decisive Checks
Four checks decide the outcome.
- Transfer of agreement. Each judge's wild agreement on non-censored English turns, with bootstrap intervals, is compared to its recomputed lab agreement. A drop of five or more points supports the claim.
- Length effect. The length coefficient should be positive in the judges' scores but zero or negative for user reactions, especially in coding and how-to tasks. That would be the naturalistic counterpart of the verbosity bias found in Justice or Prejudice?. If no cue survives control for the judge score, the bias literature's concerns do not show up in practice for this traffic, which is a useful negative result.
- Stability under abandonment. We check whether judge rankings and backend rankings stay the same across the abandonment bounds.
- Generalization beyond English. We repeat the analysis on the Chinese and Russian subsets. These are large enough, at 13% and 12% of WildChat, to detect small effects, which is the scale advantage the web-versus-lab study demonstrated.
The project succeeds if it either shows a structured, bias-explained gap between judges and real users that a multi-cue calibration narrows on held-out data, or shows convincingly that no such gap exists.
References
- Judging LLM-as-a-judge with MT-Bench and Chatbot Arena
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- Length-Controlled AlpacaEval: A Simple Debiasing of Automatic Evaluators
- WildChat: 1M ChatGPT Interaction Logs in the Wild
- Validating the web-based evaluation of NLG systems
- ObjexMT: Objective Extraction and Metacognitive Calibration for LLM-as-a-Judge under Multi-Turn Jailbreaks
- Distributional LLM-as-a-Judge