TL;DR
Current end-to-end autonomous driving systems suffer from a surprising flaw: they largely ignore global navigation commands, over-relying instead on local scene perception. To fix this, the authors introduce Sequential Navigation Guidance (SNG), a framework that combines noisy navigation paths with rich, turn-by-turn textual prompts, delivering state-of-the-art planning performance without requiring auxiliary perception tasks.
What Problem Does This Paper Address?
Most state-of-the-art end-to-end (E2E) autonomous driving models ingest simple, one-hot encoded driving commands like "Turn Left" or "Go Forward" to guide their trajectory planning. However, this paper uncovers a startling reality: corrupting or completely removing these navigation commands barely impacts the performance of popular models like Transfuser.
The authors argue that standard driving commands are fundamentally flawed due to two main issues:
- Annotation Ambiguity: Commands are often tied to fixed spatial or temporal intervals. For example, a vehicle navigating a roundabout might experience significant lateral displacement, causing a "Go Forward" intent to be erroneously labeled as a "Turn Left".
- Causal Confusion: In beyond-visual-range (BVR) scenarios, a vehicle needs to change lanes well in advance of a turn. If the global command remains "Go Forward" during the lane change, the model becomes confused by the expert trajectory's behavior.
As a result, current E2E planners simply learn to ignore global navigation inputs, leading to brittle behavior in complex, unfamiliar environments.
Methods
To bridge the gap between global routing and local planning, the authors propose the Sequential Navigation Guidance (SNG) framework, which redesigns how navigation information is represented and ingested:
- Sequential Navigation Guidance (SNG): Instead of discrete commands, SNG provides two synchronized inputs:
- Navigation Path: A reference line sampled from road centerlines up to 40 meters ahead. Crucially, the authors inject substantial noise into this path to simulate real-world localization errors and prevent the model from cheating using privileged simulator data.
- Turn-by-Turn (TBT) Information: Rich textual descriptions detailing current actions, durations, future actions, and supplementary context (e.g., "Enter the roundabout and take the third exit to the left"). These are generated automatically using a powerful offline Vision-Language Model (Qwen2.5-VL-72B).
- SNG-VLA Model Architecture: The authors build a Vision-Language-Action (VLA) model using a Qwen2.5-0.5B backbone and a SigLIP vision encoder. It processes multi-view images, TBT text, the noisy navigation path, and ego-states. To prevent the model from overfitting to ego-velocity, they implement an attention-based State Dropout Encoder (SDE). The model autoregressively outputs textual reasoning followed by trajectory waypoints via a cross-attention decoder.
Datasets
The paper utilizes and introduces the following datasets:
- NAVSIM: A real-world driving dataset used as the base for open-loop and closed-loop evaluations.
- SNG-QA Dataset (New): The authors developed an automated pipeline to generate 100,000 high-quality QA samples based on NAVSIM. The dataset breaks down driving logic into hierarchical steps: global navigation summarization, local scene planning, and trajectory waypoint generation.
- Bench2Drive: A closed-loop, multi-ability benchmark based on the CARLA simulator used to evaluate the model across various driving scenarios (merging, overtaking, traffic signs, etc.).
Key Insights and Ideas
- The Navigation Illusion: Standard E2E models are effectively "blind" to navigation commands. As shown in the command ablation experiments, feeding a model random or fixed incorrect commands results in virtually the same performance as giving it the correct command.
- Geometry + Semantics Synergy: A sparse navigation path provides excellent long-term geometric constraints, while TBT text provides immediate decision-making logic. Together, they prevent the planner from drifting or making dangerous counter-flow maneuvers.
- Less is More in Perception Supervision: Unlike many contemporary E2E models (like UniAD) that rely heavily on multi-task auxiliary losses (bounding boxes, semantic segmentation masks), SNG-VLA achieves SOTA results purely through precise navigation modeling and trajectory imitation.
Key Results
- NAVSIM Benchmark: SNG-VLA achieved a state-of-the-art PDMS (Planner Demonstration Metric Score) of 88.24, outperforming competitive baselines like DiffusionDrive (88.1) and Hydra-MDP (86.5). Notably, adding SNG to the baseline Transfuser model boosted its PDMS from 84.0 to 85.5.
- Bench2Drive Benchmark: SNG-VLA achieved a Driving Score of 67.17 and a Success Rate of 35.90%, outperforming UniAD-Base by 46.6% and 119.4% respectively.
- Efficiency: SNG-VLA operates with a highly competitive latency of 159.6 ms on an NVIDIA A6000 GPU, making it significantly faster than other large-backbone planners like ThinkTwice (762 ms) and DriveAdapter (931 ms).
Limitations and Future Work
- CARLA Evaluation Constraints: Due to real-time constraints and the lack of rich textual annotations within the CARLA simulator, the authors had to rely on rule-based methods to generate SNG inputs during Bench2Drive testing, rather than using the full SNG-QA language generation pipeline.
- Offline VLM Dependency: Constructing the SNG-QA dataset relies heavily on massive offline VLMs (Qwen2.5-VL-72B) for data curation and filtering, which is computationally expensive. Future work could explore distilling this capability into smaller, online modules.
Summary updated