SuperGlance: Driving at a Glance with a Vision-Language Model

Jieyuan Pei, Xin Yan, Mingwei Xu, Haiyi Zhang, Siyu Zou, Zhangyi Hu, Zhuoxuan Li, Yuguang Yang, Rongjunchen Zhang, Cheng Chi, Hui Xiong, Yan Wang

TL;DR

SuperGlance enables autonomous driving "at a glance" by extracting planning and reasoning directly from a Vision-Language Model's (VLM) internal hidden state in a single forward pass. By bypassing the slow, token-by-token decoding of traditional VLAs, it achieves a 6x speedup and significantly higher safety in closed-loop testing while outperforming baselines on both driving and visual reasoning benchmarks.

What Problem Does This Paper Address?

In the world of embodied AI, we often talk about System 1 (fast, instinctive reactions) and System 2 (slow, deliberate reasoning). Current Vision-Language-Action models (VLAs) for driving tend to behave like System 2: they "think" by decoding trajectories one token at a time.

The problem is that decoding is slow. A single plan can take over 500ms of GPU time. In a fast-moving car, half a second of latency means the vehicle is effectively driving on "stale" data, which leads to crashes. Previous attempts to speed this up usually moved the planning outside the VLM, losing the model's rich visual reasoning capabilities. SuperGlance aims to keep the VLM's "brain" but make its "reflexes" instantaneous.

Methods

The core innovation of SuperGlance is treating a decoder-only VLM (like Qwen3.5) as an encoder-style policy.

  1. Single-Pass Planning: Instead of generating text, the authors attach a linear Policy Head to the VLM's last hidden state at the answer position [ANS]. This head scores a fixed "vocabulary" of 5,273 candidate trajectories (derived from human driving data and rule-based maneuvers).
  2. Three-Stage Training:
    • Stage 1 (General QA): Trains the model to answer 1.16M general visual questions in a single pass. To combat "position bias" (where models just pick 'C' because it's in a certain spot), they use twin questions with shuffled options.
  3. Stage 2 (Driving QA): Focuses on driving-specific reasoning (e.g., "What is the status of the object in the red box?").
  4. Stage 3 (Policy Training): Uses Supervised Fine-Tuning (SFT) followed by Exact Reinforcement Learning. Because the trajectory set is fixed, they can compute the exact expected reward across all candidates without the noise of sampling-based RL (like GRPO).
  5. KL-Regularization: Throughout training, a Kullback–Leibler (KL) divergence term ensures the model doesn't "forget" its base language and vision capabilities while learning to drive.

Datasets

The authors evaluated SuperGlance across a broad spectrum of benchmarks:

  • Driving Benchmarks: NAVSIM (open-loop), WOD-E2E (Waymo long-tail scenes), PAI-AV, and AlpaSim (a closed-loop simulator where GPU latency is factored into the vehicle's reaction time).
  • Vision-Language Benchmarks: CV-Bench, BLINK, and 3DSRBench (spatial reasoning).
  • Language Benchmarks: MMLU, ARC-C, HellaSwag, and TruthfulQA.
  • Instruction Following: Language2Drive, which tests the model's ability to follow complex verbal commands like "Change one lane to the left and remain there."

Key Insights and Ideas

  • The Hidden State is Enough: The most significant "aha" moment is that by the time a VLM is ready to write its first token, its internal hidden state already contains enough information to both drive the car and answer complex questions about the scene.
  • Decoding is a Bottleneck, Not a Requirement: While "Chain-of-Thought" decoding helps with math or logic, it isn't strictly necessary for spatial driving tasks. SuperGlance proves that a single-pass "glance" can be just as accurate if the model is trained to compress its reasoning into that final hidden state.
  • Latency is a Safety Metric: The paper highlights that a "better" model on paper (higher open-loop accuracy) can be a "worse" driver in reality if it takes too long to compute. By reducing latency from ~500ms to 84ms, the model stays "fresher" relative to the environment.

Key Results

  • Speed: SuperGlance plans in 84 ms, making it 6.2x faster than the Qwen-Drive-1.0 baseline.
  • Safety: In closed-loop AlpaSim testing, when GPU time is charged as delay, SuperGlance drives 3.9x further per at-fault incident than the baseline.
  • Performance: It leads on 10 out of 11 visual/language benchmarks and all driving benchmarks tested.
  • Instruction Following: It successfully follows 40.5% of complex driving instructions on the Language2Drive track, whereas the baseline (without specific instruction tuning) scores near 0%.

Limitations and Future Work

  • Fixed Vocabulary: The model is limited to its 5,273 candidate trajectories. If a situation requires a highly specific maneuver not in this set (e.g., a very specific emergency swerve), the model can only pick the "least bad" option. The authors partially addressed this with rule-built candidates for braking, but a more dynamic trajectory generation might be needed.
  • Reasoning vs. Reflex: While SuperGlance excels at "at-a-glance" tasks, it trails the base VLM on complex language tasks (like MMLU) that benefit from the extra computation time afforded by multi-token decoding.
  • Future Work: The authors suggest exploring how to combine this fast "System 1" with a slower "System 2" that only kicks in when the model detects high uncertainty in its initial glance.

Summary updated

Abstract

Autonomous driving is turning to vision-language-action models (VLAs). They answer and plan by decoding, one step at a time. Each step is another forward pass, so a plan can take over a second of GPU time. Embodied AI often pairs such a slow, reasoning System 2 with a fast System 1 that acts. Human drivers plan at a glance, with System 1. Faster designs usually put System 1 outside the vision-language model (VLM), in a planner without its abilities. Yet before a VLM writes its first token, it has already read the scene into one hidden state. We show that, once trained, this hidden state is enough to drive and to answer questions about what the model sees. SuperGlance reads it to turn a decoder-only VLM into an encoder-style driving policy. Its linear head scores a fixed set of 5,273 candidate trajectories. With the set fixed, rewards are computed offline, so supervised targets can be reward-weighted and reinforcement learning maximizes the exact expected reward. Before and during policy training, multiple-choice training keeps the VLM's visual abilities in one pass, so the policy still answers questions about driving scenes. Training questions are built against shortcuts. Each must be answered right in two option orders. Against drift, a KL term keeps each stage near its starting model. One pass offers VLAs a more plausible path to System 1. SuperGlance answers visual multiple-choice questions as well as its base VLM does by decoding. It plans in 84 ms of GPU time, six times faster than Qwen-Drive-1.0, and at least as well on NAVSIM, PAI-AV and WOD-E2E. In AlpaSim closed loop, km per at-fault incident holds for both planners up to a 300-ms delay and drops beyond. With GPU time charged as delay, SuperGlance's lead over Qwen-Drive-1.0 grows from 2.8 to 3.9 times. Delayed by 1.2 s, our model falls below Qwen-Drive-1.0 at its own 518 ms.
BibTeX
@misc{pei2026superglance,
  author = {Jieyuan Pei and Xin Yan and Mingwei Xu and Haiyi Zhang and Siyu Zou and Zhangyi Hu and Zhuoxuan Li and Yuguang Yang and Rongjunchen Zhang and Cheng Chi and Hui Xiong and Yan Wang},
  title = {{SuperGlance: Driving at a Glance with a Vision-Language Model}},
  year = {2026},
  eprint = {2611.19138},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19138},
}
Figure