Deferred Exposure of Future Trajectories for Verifiable Reasoning in Autonomous Driving VLMs

Zixuan Huang, Yang Zhou, Kaixuan Wang, Guli Zhang, Hongyan Xie, Yakun Zhu, Hao Geng, Xiaozhi Chen, Yikun Ban, Deqing WangarXiv 2026 · arXiv

TL;DR

This paper identifies "trajectory anchoring bias" in autonomous driving VLMs, where models trained on ground-truth future trajectories merely rationalize known outcomes post-hoc rather than inferring decisions from scene evidence. To fix this, the authors introduce AD-MCQ, a multiple-choice trajectory benchmark, and DEFT-RLVR, a training framework that hides candidate trajectories until after the model commits to a scene-based decision. This approach significantly improves driving reasoning and decision-making while preventing the catastrophic degradation of general visual capabilities typically caused by standard fine-tuning.

What Problem Does This Paper Address?

Vision-Language-Action (VLA) models are increasingly used in autonomous driving (AD) to handle high-level reasoning via Chain-of-Thought (CoT) prompting. However, standard data annotation pipelines show the teacher model the ground-truth (GT) future trajectory before it writes the reasoning trace.

The authors discover that this creates a severe trajectory anchoring bias: instead of driving based on scene evidence, the model works backward to justify a pre-revealed outcome. This leads to post-hoc rationalizations and severe hallucinations—such as inventing traffic signs that don't exist just to explain why it decided to turn. Hiding the trajectory completely and forcing open-ended coordinate generation is also problematic, as it entangles high-level reasoning with complex, low-level geometric synthesis.

GT conditioning induces post-hoc rationalization by fabricating cues.

Methods

To solve this, the authors propose a two-part framework:

  1. AD-MCQ (Autonomous-Driving Multiple-Choice Question): Instead of generating raw coordinates, the planning task is recast as selecting from a set of discrete, scene-specific candidate trajectories generated via a K-means codebook ($K=8192$). This turns trajectory planning into an exactly verifiable decision.
  2. DEFT-RLVR (Deferred Exposure of Future Trajectories for RLVR): To prevent the model from using the multiple-choice options as a new shortcut, exposure to the candidates is deferred using a two-turn interaction:
  3. Turn 1 (Causal Decision Reasoning): The model analyzes the scene context alone (visuals, ego history, and navigation commands) and commits to a high-level decision (e.g., "decelerate to a stop").
  4. Turn 2 (Executable-Trajectory Grounding): The candidate trajectories are revealed, and the model must select the option that best matches its Turn 1 commitment.

The entire sequence is optimized jointly using Group Relative Policy Optimization (GRPO). Crucially, they use Structured Rubric Rewards: an offline vision-language model generates a scene-specific rubric, and an online text-only judge evaluates the Turn 1 reasoning trace only if the final trajectory selection is correct. This keeps the evaluation efficient and focused on text quality without visual token dilution.

The framework of AD-MCQ construction and DEFT-RLVR workflow.

Datasets

The authors leverage data from the Waymo Open E2E dataset and an internal driving corpus, pooling over 514,000 scene-trajectory pairs. They introduce:

  • AD-MCQ-500: A curated evaluation test split of 500 causally demanding, long-tail scenes (hard braking, sharp turns, stopping from motion) with structured hard distractors.
  • nuScenes: Used as an out-of-distribution (OOD) external dataset to evaluate cross-domain transfer capability.

Key Insights and Ideas

  • Information Order Matters: Revealing the answer before the rationale ruins the causal faithfulness of CoT. By simply changing when the model sees the future options, it is forced to actually "look" at the road and reason from evidence.
  • MCQ as a Verification Interface: Multiple-choice formats are often criticized in LLM evaluation, but here, casting trajectory planning as an MCQ elegantly decouples high-level causal reasoning from low-level geometric coordinate generation.
  • Avoiding the Specialization Trap: Directly fine-tuning VLMs on trajectory tokens (SFT) causes a catastrophic drop in general visual capabilities because it warps the pretrained token distribution. DEFT-RLVR actually preserves or slightly improves general visual perception because RL reinforces existing reasoning pathways rather than forcing the internalization of a new coordinate vocabulary.

Key Results

  • Driving Performance: On Qwen3-VL-8B-Instruct, DEFT-RLVR boosted MCQ accuracy from 28.1% (base) to 77.9%, outperforming the candidate-visible baseline (JEFT + RLVR) by 16.8 percentage points.
  • Causal Faithfulness: Human and automated metrics showed substantial gains in Causal Faithfulness Scores (CFS) and High-Level-Decision Consistency (HLD), proving the reasoning was genuinely grounded in scene evidence.
  • Generalization & Retention: DEFT-RLVR maintained or slightly improved scores across 12 general vision-language benchmarks (averaging ~56.09% vs 54.81% base), whereas standard imitation learning (SFT) caused general capabilities to plummet toward zero. It also successfully generalized to the out-of-distribution nuScenes dataset.

Limitations and Future Work

  • Discrete Codebook Constraints: While an 8,192-token codebook provides a low reconstruction error (~0.29m Average Displacement Error), it still quantizes continuous driving paths, which might limit fine-grained control in highly complex or dynamic environments.
  • Dependence on a Strong Judge: The framework relies on an offline/online teacher model (like Qwen3.6-35B) to generate rubrics and grade reasoning traces, introducing a dependency on external model capabilities during training.
  • Future Directions: The authors suggest exploring how to scale candidate construction to dynamically changing environments and extending this verifiable reasoning paradigm to closed-loop simulation testing.

Summary updated

Abstract

Recent Vision-Language-Action (VLA) models for autonomous driving (AD) increasingly utilize chain-of-thought (CoT) supervision to enhance the reasoning capabilities of their Vision-Language Model (VLM) components, yet existing annotation pipelines commonly expose the teacher model to the logged ground-truth (GT) future trajectory. We empirically show that this induces trajectory anchoring bias: teacher models rationalize the revealed outcome rather than infer a decision from scene evidence, producing less causally faithful CoTs and substantially more severe hallucinations, especially in causally challenging scenes. Removing the GT trajectory eliminates this shortcut, but open-ended trajectory generation entangles high-level decision-making with precise geometric synthesis and low-level dynamics. To make trajectory-level driving decisions verifiable without requiring open-ended trajectory synthesis, we introduce Autonomous-Driving Multiple-Choice Question (AD-MCQ), which casts planning as selection among explicit trajectory candidates. Taking this a step further, we propose Deferred Exposure of Future Trajectories for RLVR (DEFT-RLVR) to transform future trajectories from pre-decision anchors into post-decision verification targets. Experimental results show that DEFT-RLVR improves AD reasoning while preserving or even enhancing general visual capabilities. With VLM-only inference and controllable difficulty through candidate construction, AD-MCQ provides a flexible, scalable, and extensible foundation for future research on verifiable AD reasoning.
BibTeX
@misc{huang2026deferred,
  author = {Zixuan Huang and Yang Zhou and Kaixuan Wang and Guli Zhang and Hongyan Xie and Yakun Zhu and Hao Geng and Xiaozhi Chen and Yikun Ban and Deqing Wang},
  title = {{Deferred Exposure of Future Trajectories for Verifiable Reasoning in Autonomous Driving VLMs}},
  year = {2026},
  eprint = {2608.01755},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.01755},
}
Figure