When an LLM Judge Disagrees With Itself, Is the Traffic Scene Ambiguous or Is the Judge Noisy?

Large language models are increasingly asked to judge driving behavior. They answer questions about traffic interactions, grade simulated rollouts, and decide inside refinement loops whether a generated scene "did the right thing." Ask such a judge the same question twice and you often get different answers. The stochasticity literature treats this as a defect to be driven toward zero. Some traffic questions, though, really are uncertain. Whether a cyclist yields to a car that drifted a meter left of its recorded path has no single answer, and a good judge's spread of answers should reflect that uncertainty. For text tasks we cannot separate these two sources of disagreement cheaply, because the only reference for "how ambiguous is this?" is a small, noisy pool of human raters.

Driving simulation supplies a better reference. A reactive simulator can replay a scene many times with learned background agents that respond to what the car under test actually does, rather than replaying recorded trajectories as fixed scripts. The fraction of those replays in which a collision or a yield occurs is an objective, computable measure of how uncertain the outcome is. This proposal uses that measure as a reference ruler for LLM judges. The goal is to split a judge's run-to-run disagreement into the part that tracks real outcome uncertainty and the leftover noise. A second question is whether judges wrongly blame the car under test for "ghost" collisions, meaning collisions caused only because recorded background agents cannot react. The third step is to train an open judge whose sampled answers match the simulator's outcome distribution.

Core Claim

The claim is that repeated simulation with reactive background agents gives a free, scalable ground truth for the distribution of outcomes. Against that ground truth, the stability of LLM judges on physical multi-agent questions can be measured more meaningfully than with consistency-only metrics. Two predictions follow. First, current judges will show substantial disagreement that is uncorrelated with real ambiguity, which means both temperature-zero decoding and majority voting are measuring the wrong thing. Second, a judge fine-tuned to match simulator outcome frequencies will be better calibrated on held-out scene types than one trained on majority labels, without losing top-1 accuracy.

This differs from each neighboring line of work. Are Your LLMs Capable of Stable Reasoning? introduced G-Pass@k, which scores how often a model gives the same correct answer across repeated samples. On math, where there is one right answer, it shows that general models lose more than half their accuracy when full consistency is required. G-Pass@k assumes every question has a single answer, so it penalizes a judge that correctly splits 60/40 on a genuinely 60/40 scene. Distributional LLM-as-a-Judge trains judges to match human rater distributions and reports SNLI divergence of 0.23 against 0.60 for single-label training. It supplies the training method we reuse, but its target distribution comes from humans on static text. Our target comes from a simulator on physical interactions. REQUAL-LM averages many samples to suppress variance. That is the opposite design choice to ours, and it serves as a baseline.

Research Program

The first component builds the reference labels. Scenes come from the Waymo Open Motion Dataset, the basis of The Waymo Open Sim Agents Challenge. In each scene we perturb the car under test, called the ego vehicle, away from its logged path by lateral offsets, speed changes, and delayed braking. Two things are recorded for each perturbed scene.

The first record is the outcome under non-reactive log replay, where background agents follow their recorded paths regardless of the ego's deviation. The second is the outcome across $K=32$ replays in which the background agents are reactive. Thirty-two matches the challenge's sampling protocol, and it bounds the standard error of the estimated frequency near 0.09 in the worst case, since the error of a binomial proportion is at most $\sqrt{0.25/K}$. We use two unrelated reactive models so that the labels do not reflect one model's habits:

  • ProSim, a promptable closed-loop simulator that runs 64 agents per 50 ms step on one GPU;
  • the next-token motion model from Do LLM Modules Generalize?, which reports a realism meta-metric of 0.778.

For each scene and event (collision within 3 s, or background agent yields), the label is the event frequency $q(x)$ across reactive replays. A scene is a ghost collision if it collides under log replay but $q(x)$ is near zero.

The second component probes judges. Following STSBench, which found that text-only GPT-4o given coordinates (57.08%) beats driving vision-language models given images (29.33% to 39.51%), judges receive structured coordinate histories rather than images. Each judge answers the event question and a fault question $N$ times at its default temperature and $N$ times at temperature zero. Temperature-zero repeats capture the hardware-level drift described in Towards large language model-based personal agents in the enterprise. Let $\hat p(x)$ be the judge's yes-frequency. We score calibration and variance tracking separately:

$$\mathrm{CalErr}=\mathbb{E}_x\,|\hat p(x)-q(x)|,\qquad \rho_{\mathrm{amb}}=\mathrm{corr}\big(\hat p(x)(1-\hat p(x)),\;q(x)(1-q(x))\big).$$

Here $\rho_{\mathrm{amb}}$ is high when the judge wavers mostly on scenes where the simulator also wavers. The residual judge variance on scenes with $q$ near 0 or 1 is pure noise.

The third component trains a judge on simulator frequencies. An open model is fine-tuned from Qwen2.5 or LLaMA3.1 with the divergence-plus-cross-entropy objective of Distributional LLM-as-a-Judge, using $q(x)$ from one reactive simulator as the target. It is evaluated on scene categories held out from training and against labels from the other simulator.

Baselines

The naive reference is the log-replay verdict itself: any collision under replay counts as ego fault. This is what geometric collision checks do in replay-based evaluation, and it defines the ghost-collision problem the judges must beat.

The strongest non-LLM competitor is a geometric regressor. It is a gradient-boosted model trained on time-to-collision, closest-approach distance, relative heading, and speed to predict $q(x)$. It sees the same training scenes, uses the same held-out split, and gets the same hyperparameter search budget as the fine-tuned judge. It is the most dangerous baseline because much of collision probability is kinematic.

The strongest existing LLM configuration is GPT-4o with the coordinate-input prompting used in STSBench. The reported reference number there is 57.08% multiple-choice accuracy. There is no reported result found on Lacuna for any judge's calibration against reactive-rollout outcome frequencies, so our first experiment establishes that number.

Two aggregation baselines each get the same total number of calls as the distributional judge's $N$ samples: majority vote over those samples, and the embedding-centroid selection of REQUAL-LM. The key ablation is the same open model fine-tuned on majority labels (whether $q>0.5$) instead of $q$. This isolates whether the distributional target, rather than simulator data alone, drives any gain.

Beating the baselines means two things. The fine-tuned judge must have lower CalErr and higher $\rho_{\mathrm{amb}}$ than GPT-4o and the majority-label ablation, with non-overlapping bootstrap intervals. On the subset of scenes where the geometric regressor is least confident, it must match or beat that regressor's CalErr.

Why not just…?

Why not just run the counterfactual resimulation and skip the judge? For attributing a collision to its cause in simulation, resimulation is the better tool. That is exactly why we use it as the reference ruler rather than as a competitor. Judges still matter wherever resimulation is not being run: in refinement loops like Controllable Traffic Simulation through LLM-Guided Hierarchical Reasoning and Refinement, which accepts a single LLM verdict as ground truth, and in language-level scene questions like STSBench. The contribution is to measure whether those verdicts can be trusted and to repair them. It is not to replace simulation.

Why not just set temperature to zero? Greedy decoding hides instability rather than removing it, as the G-Pass@k results show. Hardware nondeterminism persists even at zero temperature. More fundamentally, a point answer cannot express a 60/40 scene. Our temperature-zero repeats measure the leftover drift directly, and CalErr shows the cost of collapsing to one answer on scenes with intermediate $q$.

Why not just take a majority vote? Voting also collapses to a point, and it averages correlated errors. The reference results are REQUAL-LM's unweighted aggregation staying at a 0.66 female-to-male ratio, and the G-Pass@k paper's observation about voting over flawed reasoning. The matched-budget voting baseline tests this directly.

Why not just use time-to-collision features? This alternative is a real threat. If the geometric regressor matches the fine-tuned judge everywhere, then LLM judges add nothing for predicting physical outcomes, and we will report that. The hypothesis is that judges help on yield and negotiation events, where outcomes depend on right-of-way semantics rather than kinematics. The per-event-type comparison settles it.

Decisive Checks

The first check is whether off-the-shelf judges' $\rho_{\mathrm{amb}}$ is near zero while their disagreement rate on scenes with $q\in\{0,1\}$ stays high. That result would show that consistency metrics and ambiguity are confounded in current judges.

The second is the ghost-collision rate: the fraction of replay-only collisions that each judge blames on the ego. A judge that cannot tell non-reactive artifacts apart from real faults should not sit in an evaluation loop.

The third is transfer. The fine-tuned judge's calibration must hold against labels from the second reactive simulator it never trained on, and it must not reduce accuracy on the 971 human-verified STSBench questions. It should also narrow the 20–30% gap that STSBench reports for agent-to-agent questions.

The fourth check guards against circularity. On a 300-scene subset, human annotators judge fault in the ghost-collision cases. If humans disagree with the simulator labels, the reference itself is suspect and the claims shrink accordingly.