Vision-language models that answer questions about driving video mostly fail because they misread the scene, not because they reason badly. When STSBench gave GPT-4o ground-truth 3D trajectories as text, it scored 57.08% on its human-verified driving questions. Driving-specialist models that had to read the camera frames themselves scored 39.51% (DriveMM) and 29.33% (OmniDrive). Part of the problem is that the camera itself moves. A car that looks like it slides left in the image may be standing still while the ego vehicle turns right. The usual fix is more carefully labeled real video, and labeling is the bottleneck: STSBench needed a human to look at every mined question, about 14.5 seconds each, to end up with 971 items.
This note proposes moving most of that human work into a simulator, where it becomes free, and paying humans only at the one step where simulation still breaks labels. A driving simulator knows every agent's true 3D path and the exact ego-vehicle motion. It can therefore produce question–answer pairs with guaranteed-correct answers. It can also produce ego-motion minimal pairs: the same traffic scene recorded twice with two different ego trajectories. In such a pair, questions about how agents move in the world keep their answer, and questions about how agents move relative to the ego vehicle flip theirs. The weak link is the step that makes simulator frames look real. A diffusion translator can erase a pedestrian or recolor a traffic light, which silently invalidates a correct label. We therefore check each translated clip against the simulator's own ground truth and send only clips that fail or are ambiguous to a human.
Core Claim
Two claims are tested. First, a vision-language model fine-tuned on translated simulator clips organized as ego-motion minimal pairs will answer real driving questions better than the same model fine-tuned on the same number of automatically mined real clips. The gain should concentrate on the Agent-to-Agent and Agent-to-Ego categories and on items where the ego vehicle is moving. Second, a geometric "did the label survive translation?" check can direct a small human budget at the right clips. With that check, near-full-verification quality should cost a small fraction of the human seconds that verifying everything would cost.
What is new is the combination of two things. The minimal pair is built along the ego-motion axis, which is the specific shortcut moving cameras invite. Human verification is aimed at translation failures rather than at the labels themselves.
Relation to Prior Work
Several papers motivate the proposal. STSBench mines 43 scenario types from nuScenes tracks, maps and ego odometry and verifies every item by hand. It supplies our real test set and our question templates, but it never uses simulation. BEAR attributes 54.8% of embodied model failures to perception and observes that models struggle to map egocentric camera motion to world-frame layout. MIRAGE reports that scaling barely reduces spatial hallucinations. Together these suggest that more general pretraining will not fix the problem on its own.
Other papers supply methods we reuse. Driving with DINO translates CARLA video toward nuPlan appearance by conditioning a video diffusion model on DINOv3 features. It removes high-frequency, simulator-specific detail with a projection onto the main directions of variation (PCA) and reports sKID 9.30 and sFID 21.62. Those scores measure image realism only. The paper does not test whether answer-relevant objects survive translation, and that is exactly the gap our check fills. Hartmann et al. show that minimal pairs, examples that differ in one cue but carry different labels, expose shortcuts, but in text they require expert hand-editing. Simulation makes ego-motion minimal pairs free.
Two papers constrain the design. Bayesian Active Learning for Sim-to-Real Robotic Perception uses epistemic uncertainty, the model's uncertainty about its own parameters, to choose which real images humans label. It reports matching full supervision with 36% of the data on EDAN, but it degrades under heavy class imbalance on YCBV and only covers static detection. We keep their uncertainty routing as a competing baseline rather than assuming it transfers. ReBot shows that replaying physically grounded trajectories and compositing them onto real backgrounds avoids generative hallucination, but only for stationary tabletop cameras. Under ego motion the background itself moves, so plain compositing does not apply directly. That is why we accept a diffusion translator and add a check after it.
FlowAD builds ego velocity and steering into the driving model's architecture, and Adeli et al. separate global from local pedestrian motion. Both handle ego motion inside the architecture. We instead handle it through the training data, with an unchanged vision-language model, so the two approaches are complementary.
Research Program
The first component is a simulator question generator. STSBench's scenario miners need only 3D tracks, a map and ego odometry. We re-implement them on CARLA logs, including the Bench2Drive scenarios used by FlowAD, so simulated and real questions share templates and hard-negative options. Train and test then differ in visual domain, not in question format.
For each scene we replay the non-ego agents exactly while driving the ego vehicle along two or more trajectories that differ in speed, lane or turning. Each question is tagged by whether its answer is invariant or variant under the ego change. A pair counts as a minimal pair only when the variant answer actually flips.
The second component translates every clip to realistic appearance with a DwD-style translator. The third is the label-survival check. We project the simulator's 3D boxes into each translated frame using the known camera parameters and run an off-the-shelf detector and tracker on the translated frames. A clip passes only if every agent referenced by the question, or by its correct answer, is detected and associated in the frames the question depends on. For attribute questions such as light state, an attribute classifier must agree with the simulator's value.
Clips that clearly fail are discarded. Clips in an uncertain band near the matching thresholds go to a human with an STSBench-style interface. A uniform random sample of passed and discarded clips is also human-audited, so we can measure how accurate the check is instead of assuming it. Finally, we fine-tune InternVL 2.5 8B, the strongest raw-pixel model reported on STSnu in the file (46.07%), with a consistency term added to the usual loss. This term penalizes disagreement between the two members of a pair on invariant questions and rewards a correct flip on variant ones.
Baselines
The naive reference point is InternVL 2.5 8B with no fine-tuning, which STSBench reports at 46.07% on STSnu. We re-run it under our prompting so that all numbers share one setup. DriveMM (39.51%) and OmniDrive (29.33%) are quoted as reported. The text-only GPT-4o result with ground-truth trajectories (57.08%) is not a competitor, because it is given perfect perception. It serves as a rough ceiling on what better perception could buy.
The strongest competitor is the same model fine-tuned on real nuScenes clips. Their questions are mined automatically by STSBench's pipeline without human filtering, and every scene that appears in STSnu is excluded. This baseline has no domain gap, and its only weakness is label noise from occlusion and visibility errors. It gets the same number of question instances, the same steps, the same learning-rate sweep and the same compute as our method. We beat it only if we exceed it on overall STSnu accuracy and on the Agent-to-Agent category, with bootstrap confidence intervals that do not overlap. STSnu is small, so we report intervals for every number. A second variant of this baseline gets a human-verification budget equal to ours, spent on random real items.
Three ablations isolate our ingredients. The first keeps the translated simulator data but drops the pairing, sampling single ego trajectories at the same count; this tests the minimal pairs. The second keeps the pairs but drops the label-survival check and trains on all translated clips; this tests whether the check matters. The third replaces the geometric check with uncertainty-based routing in the style of Bayesian active learning, at equal human seconds; this tests whether simulator ground truth beats model uncertainty as the routing signal.
Why not just…?
Why not just fine-tune on automatically mined real nuScenes questions and skip simulation? This is the most serious threat. Real clips have no appearance gap, and mining noise may matter less than we expect. The first baseline above settles it directly. If it matches us overall and on moving-ego items, the minimal-pair idea gives no benefit, and we would report that. Our reason to expect otherwise is that real data cannot hold traffic fixed while changing only the ego trajectory. The model can therefore never see a pure contrast between apparent and world motion.
Why not just train on raw CARLA renders with appearance randomization? The pairing might be what matters, with realism as a costly extra. We test this by adding an untranslated-pairs condition to the ablations. If it matches the translated version, the translation and the human budget are unnecessary. The proposal would then shrink to "minimal pairs from simulation," which is still a usable result.
Why not just give the model tracks and odometry from a perception stack, the way BEAR-Agent uses external tools? BEAR reports a 17.5% relative gain for GPT-5 with tool-based grounding, so this route is real. However, it changes the question being asked: it measures whether the tracker is good, not whether the vision-language model perceives. We include a tool pipeline that feeds predicted nuScenes tracks to the same model as text, as a reference. We treat our method as a way to improve the model's own perception, and we report whether its gains add to the tool pipeline's.
Why not just discard every flagged clip and use no humans at all? This may be enough. The random human audit measures how often the check passes broken labels and rejects good ones. If accuracy with automatic filtering alone lies within the confidence interval of accuracy with human review of the uncertain band, we will say humans are only needed for auditing.
Proposed Experiments
The first decisive check is the main comparison on STSnu: overall accuracy and accuracy in the Ego, Agent, Agent-to-Ego and Agent-to-Agent categories, for our method against every baseline and ablation, at matched data and compute. The second is an ego-motion interference measure on real data. Using nuScenes odometry, we split STSnu into items where the ego vehicle is stationary or slow and items where it moves or turns, and report the accuracy gap between the two. Our method should shrink that gap relative to the real-data baseline; if it raises both splits equally, the pairs are not doing what we claim. On held-out translated pairs we also report pair consistency, meaning the rate of correct invariance and correct flips.
The third check is a cost curve: STSnu accuracy against total human seconds spent. It compares geometric routing, uncertainty routing and random verification, with STSBench's 14.5 seconds per item as the unit cost of full verification. The label-survival check succeeds if it reaches the accuracy of verifying everything with a clearly smaller share of human time than uncertainty routing needs. The audit sample also reports how often translation actually destroyed answer-relevant evidence. DwD does not report this figure, and it is worth knowing on its own.
References
- STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving
- BEAR: Dissecting Embodied Abilities in Multimodal Language Models through Skill-level Evaluation and Diagnosis
- MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLM
- Driving with DINO: Vision Foundation Features as a Unified Bridge for Sim-to-Real Generation in Autonomous Driving
- A Multilingual Benchmark for Probing Negation-Awareness with Minimal Pairs
- Bayesian Active Learning for Sim-to-Real Robotic Perception
- ReBot: Scaling Robot Learning with Real-to-Sim-to-Real Robotic Video Synthesis
- FlowAD: Ego-Scene Interactive Modeling for Autonomous Driving
- Disentangling Human Dynamics for Pedestrian Locomotion Forecasting with Noisy Supervision