TL;DR
SuperGlance enables autonomous driving "at a glance" by extracting planning and reasoning directly from a Vision-Language Model's (VLM) internal hidden state in a single forward pass. By bypassing the slow, token-by-token decoding of traditional VLAs, it achieves a 6x speedup and significantly higher safety in closed-loop testing while outperforming baselines on both driving and visual reasoning benchmarks.
What Problem Does This Paper Address?
In the world of embodied AI, we often talk about System 1 (fast, instinctive reactions) and System 2 (slow, deliberate reasoning). Current Vision-Language-Action models (VLAs) for driving tend to behave like System 2: they "think" by decoding trajectories one token at a time.
The problem is that decoding is slow. A single plan can take over 500ms of GPU time. In a fast-moving car, half a second of latency means the vehicle is effectively driving on "stale" data, which leads to crashes. Previous attempts to speed this up usually moved the planning outside the VLM, losing the model's rich visual reasoning capabilities. SuperGlance aims to keep the VLM's "brain" but make its "reflexes" instantaneous.
Methods
The core innovation of SuperGlance is treating a decoder-only VLM (like Qwen3.5) as an encoder-style policy.
- Single-Pass Planning: Instead of generating text, the authors attach a linear Policy Head to the VLM's last hidden state at the answer position
[ANS]. This head scores a fixed "vocabulary" of 5,273 candidate trajectories (derived from human driving data and rule-based maneuvers). - Three-Stage Training:
- Stage 1 (General QA): Trains the model to answer 1.16M general visual questions in a single pass. To combat "position bias" (where models just pick 'C' because it's in a certain spot), they use twin questions with shuffled options.
- Stage 2 (Driving QA): Focuses on driving-specific reasoning (e.g., "What is the status of the object in the red box?").
- Stage 3 (Policy Training): Uses Supervised Fine-Tuning (SFT) followed by Exact Reinforcement Learning. Because the trajectory set is fixed, they can compute the exact expected reward across all candidates without the noise of sampling-based RL (like GRPO).
- KL-Regularization: Throughout training, a Kullback–Leibler (KL) divergence term ensures the model doesn't "forget" its base language and vision capabilities while learning to drive.
Datasets
The authors evaluated SuperGlance across a broad spectrum of benchmarks:
- Driving Benchmarks: NAVSIM (open-loop), WOD-E2E (Waymo long-tail scenes), PAI-AV, and AlpaSim (a closed-loop simulator where GPU latency is factored into the vehicle's reaction time).
- Vision-Language Benchmarks: CV-Bench, BLINK, and 3DSRBench (spatial reasoning).
- Language Benchmarks: MMLU, ARC-C, HellaSwag, and TruthfulQA.
- Instruction Following: Language2Drive, which tests the model's ability to follow complex verbal commands like "Change one lane to the left and remain there."
Key Insights and Ideas
- The Hidden State is Enough: The most significant "aha" moment is that by the time a VLM is ready to write its first token, its internal hidden state already contains enough information to both drive the car and answer complex questions about the scene.
- Decoding is a Bottleneck, Not a Requirement: While "Chain-of-Thought" decoding helps with math or logic, it isn't strictly necessary for spatial driving tasks. SuperGlance proves that a single-pass "glance" can be just as accurate if the model is trained to compress its reasoning into that final hidden state.
- Latency is a Safety Metric: The paper highlights that a "better" model on paper (higher open-loop accuracy) can be a "worse" driver in reality if it takes too long to compute. By reducing latency from ~500ms to 84ms, the model stays "fresher" relative to the environment.
Key Results
- Speed: SuperGlance plans in 84 ms, making it 6.2x faster than the Qwen-Drive-1.0 baseline.
- Safety: In closed-loop AlpaSim testing, when GPU time is charged as delay, SuperGlance drives 3.9x further per at-fault incident than the baseline.
- Performance: It leads on 10 out of 11 visual/language benchmarks and all driving benchmarks tested.
- Instruction Following: It successfully follows 40.5% of complex driving instructions on the Language2Drive track, whereas the baseline (without specific instruction tuning) scores near 0%.
Limitations and Future Work
- Fixed Vocabulary: The model is limited to its 5,273 candidate trajectories. If a situation requires a highly specific maneuver not in this set (e.g., a very specific emergency swerve), the model can only pick the "least bad" option. The authors partially addressed this with rule-built candidates for braking, but a more dynamic trajectory generation might be needed.
- Reasoning vs. Reflex: While SuperGlance excels at "at-a-glance" tasks, it trails the base VLM on complex language tasks (like MMLU) that benefit from the extra computation time afforded by multi-token decoding.
- Future Work: The authors suggest exploring how to combine this fast "System 1" with a slower "System 2" that only kicks in when the model detects high uncertainty in its initial glance.
Summary updated