Judges That Must Look: Image-Swapped Training for Grading Rare-Hazard Driving Reasoning

When a vision-language model explains a driving decision ("slow down: a deer is crossing ahead in the right lane"), we increasingly rely on another model to grade that explanation. Such a grader is called a "judge," meaning a multimodal model prompted or trained to score other models' outputs. The trouble is that judges hallucinate too. On rare road hazards, which are almost absent from training data, a judge can reward an explanation because it sounds like correct safety reasoning, without checking whether the deer is actually in the image or in the lane. This proposal builds judges that cannot pass their training by reading the text alone. We use controllable synthetic image editing to make pairs of nearly identical driving scenes in which the same explanation is correct for one image and hallucinated for the other. We then train a compact judge to give opposite verdicts to the same text across the pair.

The result would be a grader for long-tail driving reasoning whose verdicts depend measurably on the pixels. It would also give a diagnostic, the paired flip rate, that exposes how much existing judges are really grading prose. This matters because current long-tail driving benchmarks score only the final action, so a planner that brakes for the right reason and one that brakes because of an invented obstacle get the same credit.

Core Claim

Existing ways of bootstrapping training data for judges create flawed responses by editing the text while the image stays fixed. A judge trained that way can separate good from bad responses using textual cues such as internal inconsistency, generator style, or missing detail. Our claim is that for rare-hazard driving scenes, most of the judging signal that text-edited data teaches is not visual. Supervision in which the text is held fixed and the image is swapped, built from minimally different synthetic scenes, is the ingredient that makes a judge's verdicts track visual evidence. This should hold most strongly on hazard types the judge never saw during training. The claim can be falsified. If text-flaw bootstrapping alone produces judges whose verdicts already flip correctly when the image is swapped, the proposal is unnecessary.

Where This Sits

The work draws on three lines of prior work, each playing a different role.

The first line supplies the bootstrapping recipe we modify. VideoJudge takes gold video descriptions, injects controlled flaws and hallucinations into them to create lower-rated responses, and trains 3B and 7B judges that reach 98.6% pairwise accuracy on its own benchmark and 93.67% on human-verified hard cases. It works on generic video QA, and every flaw lives in the text. Halu-J shows that a 7B text judge trained on structured evidence critiques beats GPT-4o on ME-FEVER (91% vs. 83%), but it never sees images. We reuse VideoJudge's recipe of training a compact judge on generated flaws. We add image-side contrast, which neither paper has.

The second line motivates the proposal by showing the judge's weak grounding. MLLM-as-a-Judge finds that multimodal judges do better when given text descriptions than raw pixels. It also finds that step-by-step reasoning can make judges worse, because they invent visual details and then grade against them. MERLIM shows that when an object is inpainted out of an image, models still often claim it is present, guessing from scene context. POPE traces object hallucination to co-occurrence priors. Together these predict that a judge will accept "a hazard is present" whenever the scene gist suggests one.

The third line supplies the data and the regime. CoLT-Drive inserts 50 kinds of rare obstacles into 29 base driving scenes at controlled positions, producing 3,536 counterfactual samples in naturalistic-traffic and traffic-cleared versions. It evaluates planners only by exact match on meta-actions such as "slow down" or "nudge left." It also documents the SFT trap: ordinary fine-tuning on driving logs lowers pair accuracy on rare hazards from 50.3% to 32.4%. Its knowledge-preserving adaptation method, which merges fine-tuned weights back into the base model and routes between specialized adapters, reaches 60.8%.

SymMPO supplies the pairing trick. It builds symmetric preference pairs across similar images by swapping which response is preferred, which forces a policy model to consult the image. We move that idea from the policy to the judge and from generic objects to action-relevant hazards. DRIV-EX and AutoDrive-P³ show that driving models give unfaithful rationales or hallucinate unobserved objects in their reasoning chains even when their actions are good. That is exactly the failure that grading by action alone cannot see.

Research Program

Paired scenes. For each CoLT-Drive base scene we create edited pairs in which both members are edited, so insertion artifacts alone cannot distinguish them. CoLT-Drive itself warns about such generative artifacts. The pairs vary one action-relevant factor at a time. The first pair type places the hazard in the ego lane versus on the shoulder. The second places hazard A versus a visually similar but action-different hazard B, for example a cardboard box versus a concrete block. The third swaps naturalistic traffic for a cleared background, changing whether a lateral nudge is feasible. Each pair has a known correct meta-action and a known set of true visual facts, inherited from the controlled insertion.

Responses. Responses come from real planners of several sizes run on these scenes: the base Qwen3-VL-2B, its fine-tuned version, the knowledge-preserving adaptation variant, and larger open models. We also add VideoJudge-style text-edited versions of their outputs. A response is labeled correct only if its action is right and its stated visual premises hold in that image. This separates "right action, hallucinated reason" from genuinely grounded answers.

Training. The judge is a compact open multimodal model at the 3B to 7B scale used by VideoJudge. It is trained with the usual pointwise and pairwise objectives on text-flawed data, plus an image-swap term. Let $s_\theta(y \mid m)$ be the judge's score for response $y$ given image $m$. Let $y$ be correct for $m$ and hallucinated for its partner $m'$, and let $y'$ be the reverse. The added loss is

$$ \mathcal{L}_{\text{swap}} = -\log \sigma\big(s_\theta(y \mid m) - s_\theta(y \mid m')\big) - \log \sigma\big(s_\theta(y' \mid m') - s_\theta(y' \mid m)\big), $$

where $\sigma$ is the logistic function. Because each pair of terms compares the same text under two images, any judge that ignores the image gets zero margin. It therefore pays the maximal loss, $2\log 2$.

Diagnostic. The paired flip rate is the fraction of image-swapped pairs on which the judge's verdict on a fixed response changes in the correct direction. By construction a blind judge scores 0. It is the main measure of whether a judge is looking.

Generalization split. We hold out 15 of the 50 hazard types and a subset of base scenes entirely, both from pair construction and from training. The long-tail question is whether grounding transfers to rare hazards the judge has never seen, not just to seen hazards in new positions.

Baselines

The naive reference point is the current practice in this domain: exact-match scoring of the planner's meta-action, as in CoLT-Drive. It is computed on the same human-labeled test set. We report how often it marks as correct a response whose rationale is hallucinated. Beating it means agreeing better with human labels that account for rationales.

The strongest existing method is a VideoJudge-style judge (VideoJudge). It uses the same backbone and the same number of training examples as our judge, trained on text-flawed responses generated from our driving scenes, with or without its instance-specific rubric generation step. It is the strongest competitor because it is the only method in the file that trains compact multimodal judges from synthetic flaws, and its reported hard-case accuracy is high (93.67% pairwise on VJ-H). No reported result was found on Lacuna for any judge on driving-hazard reasoning, so its in-domain number must be measured here. Our judge and this baseline differ only in whether the image-swap term and pairs are included. This comparison is therefore also the ablation that isolates the key ingredient. We additionally run a matched-size control that adds the same number of extra text-flawed examples instead of image-swap pairs. This rules out the possibility that the gain comes from more data.

The frontier zero-shot baseline is a large proprietary multimodal judge prompted with a driving rubric in pairwise mode. The file's reference point is roughly 79% human agreement for GPT-4V in pairwise comparison on general images, reported by MLLM-as-a-Judge. We run it with and without step-by-step reasoning, because that paper found such reasoning can hurt.

The text-pipeline baseline is a strong text-only language model judge that receives an object-detector and tracker description of the scene instead of pixels.

All trained judges get the same tuning budget (the same hyperparameter sweep and number of steps). All are selected on a validation split drawn from seen hazard types only.

Beating the field means three things on unseen hazard types. The first is higher human agreement than the VideoJudge-style judge. The second is a paired flip rate far above that baseline. The third is no loss on general judging benchmarks.

Why not just…?

Why not just use VideoJudge-style text-flaw bootstrapping, which already includes injected hallucinations? This is the real threat. Its injected hallucinations are textual edits of a gold description, so a judge can catch many of them without looking, from contradictions or generator style. Nothing in VideoJudge measures whether its judge's verdicts change when the image changes. Our main experiment compares the paired flip rate of the matched baseline and our judge. If the baseline already flips correctly on most pairs involving unseen hazards, we will report that image-swap supervision adds nothing and drop the claim.

Why not just give a text LLM a detector-generated scene description? STSBench shows that text-only GPT-4o given ground-truth 3D trajectories reaches 57.08% and beats pixel-based models. This route is attractive for agent interactions. But rare hazards are precisely what closed-vocabulary detectors mislabel or miss, and VideoJudge found that text summaries lose dynamic detail. We test the text route directly, using a realistic detector rather than ground truth. We expect it to do well on seen object classes and to fail on unseen hazard types. If it does not fail there, a simpler pipeline wins, and we will say so.

Why not just score the action and skip the judge? Exact match cannot see a correct brake justified by an invented obstacle. Both DRIV-EX and AutoDrive-P³ report that reasoning chains in driving models contain unfaithful or hallucinated content. We measure the fraction of action-correct planner outputs with false visual premises on CoLT-Drive scenes. If that fraction is negligible, a judge adds little over exact match.

Why not just prompt a frontier judge with a careful rubric? MLLM-as-a-Judge reports modest pointwise correlation with humans (Pearson 0.490 for GPT-4V), scores that cluster at the top, and reasoning chains that invent visual premises. Compact trained judges have matched or beaten much larger zero-shot ones in Halu-J and VideoJudge. We still run this baseline, including its paired flip rate. A large model may look more than a small one, and that would be a useful finding on its own.

Proposed Experiments

Test set. We build a human-labeled test set of roughly 1,500 planner responses on CoLT-Drive-derived scenes. Each response is labeled for action correctness and for the truth of every stated visual premise, and the set is balanced across seen and unseen hazard types. STSBench's human-in-the-loop verification took about 14.5 seconds per sample, which suggests this is affordable.

Primary metrics. These are pairwise agreement with human preference, pointwise mean absolute error against human ratings, rationale-hallucination detection recall, and the paired flip rate, all reported separately for seen and unseen hazards.

Decisive check. The decisive check is the matched comparison against the VideoJudge-style judge on unseen hazard types. The proposal succeeds only if image-swap training raises the paired flip rate substantially and improves human agreement. A higher flip rate with no gain in human agreement would mean the judge learned to be sensitive to images without learning to be right.

Artifact control. We evaluate on pairs in which the hazard is inserted identically in both images but only a background agent changes. A judge that exploits insertion artifacts rather than grounding should fail this control.

Retention. We run all trained judges on the MLLM-as-a-Judge benchmark and VideoJudge's human-verified hard set (VJ-H). The CoLT-Drive SFT trap warns that specializing to driving can erase general ability, and we need to show the judge has not narrowed.

Downstream test. As a secondary check, we use the judge as a reward to fine-tune the 2B planner on CoLT-Drive. The comparison point is knowledge-preserving adaptation's 60.8% pair accuracy, together with its 79.1% on the DriveQA-T driving-exam benchmark. We ask whether a grounded-rationale reward reduces hallucinated premises without falling into the SFT trap. We treat this as evidence of usefulness, not as the core claim.