VLGA: Vision-Language-Geometry-Action Models for Autonomous Driving

Jin Yao, Dhruva Dixith Kurra, Tom Lampo, Zezhou Cheng, Danhua Guo, Burhan YamanarXiv 2026 · arXiv

TL;DR

VLGA introduces a dedicated, parameter-isolated geometry expert into Vision-Language-Action (VLA) models for autonomous driving, supervised by a dense per-pixel pointmap reconstruction loss against LiDAR. By combining language reasoning with dense 3D spatial grounding, VLGA achieves state-of-the-art performance on both open-loop (nuScenes) and closed-loop (Bench2Drive) autonomous driving benchmarks, significantly reducing long-horizon collision rates.

What Problem Does This Paper Address?

End-to-end Vision-Language-Action (VLA) models excel at high-level scene understanding and language reasoning, but they struggle with the continuous spatial precision required for safe trajectory planning. Driving is inherently a 3D spatial task, and language tokens alone cannot accurately capture dense geometric structures.

Existing approaches to ground VLA models in 3D space fall into three flawed paradigms:

  1. Sparse 3D Perception: They use bounding boxes or occupancy grids, which lack dense spatial grounding.
  2. Feature Injection: They inject dense 3D features directly into the Large Language Model (LLM) stream, which forces the same parameters to handle both text and geometry without a dedicated parameter budget or explicit geometric objective.
  3. Geometry-Only Models: They achieve dense spatial grounding but drop the language reasoning stream entirely.

VLGA bridges this gap by asking: Can we give dense 3D geometry its own dedicated parameter budget within a VLA policy while keeping the language stream completely intact?

Paradigms for grounding driving policies in 3D geometry.

Methods

The authors propose VLGA (Vision-Language-Geometry-Action), a four-expert driving policy built on a Mixture-of-Transformers (MoT) backbone.

The Four-Expert Architecture

VLGA processes multi-view camera inputs, navigation instructions, and ego status using four specialized experts coupled via masked joint attention:

  • Understanding Expert (U): Handles language and scene semantics.
  • Perception Expert (P): Processes sparse agent, map, and occupancy queries.
  • Geometry Expert (G): Processes dense spatial structures using features from a pretrained, frozen geometry backbone (DVGT-2).
  • Action Expert (A): Emits the final trajectory via flow matching, conditioning its predictions on the other three streams.
VLGA architecture overview.

Dense Pointmap Supervision

To ensure the geometry stream actually encodes task-relevant 3D structures, the authors introduce a per-pixel pointmap regression loss against ground-truth LiDAR data during training. A lightweight transformer decoder predicts 3D points in the ego LiDAR frame alongside an uncertainty logit. Crucially, no LiDAR is required at inference time; this decoder is discarded after training.

Two-Stage Training Schedule

To prevent the geometric loss from interfering with the action expert's optimization trajectory, training is split into two stages:

  1. Geometry Stage: The action expert and inherited streams are frozen. Only the geometry expert, projector, and pointmap decoder are trained using the pointmap loss.
  2. Joint Stage: The action expert is unfrozen, and the model is trained on a combined loss of action (flow matching) and a reduced-weight pointmap loss.

Datasets

VLGA was trained and evaluated on two prominent autonomous driving benchmarks:

  • nuScenes: Used for open-loop trajectory planning evaluation, featuring 6,019 validation samples from 150 urban driving scenes.
  • Bench2Drive: A closed-loop driving benchmark containing 220 driving routes across 12 CARLA simulator towns, evaluating the model under active closed-loop control.

Key Insights and Ideas

  • Dedicated Parameter Budgets Matter: Isolating the geometry expert within the MoT architecture allows the model to learn dense spatial representations without diluting the LLM's language capabilities.
  • Supervision Over Mere Injection: Simply feeding 3D features into a model isn't enough. Explicitly forcing the geometry stream to reconstruct the 3D world via pointmap regression ensures the network retains critical spatial details.
  • Safety-Critical Gains: The empirical benefits of dense geometry are concentrated on safety-critical, long-horizon metrics (like avoiding collisions 3 seconds out) and spatial-precision-demanding scenarios (like "Give Way" and "Emergency Braking"), rather than just improving average trajectory displacement (L2 error).

Key Results

Open-Loop Evaluation (nuScenes)

In the leakage-free "without ego status" setting, VLGA-Large outperformed all counterpart VLA methods on 15 out of 16 planning metrics:

  • Achieved the lowest ST-P3 L2 average error of 0.50 m.
  • Recorded a 3-second collision rate of 0.18%, which is roughly 2.5–3× lower than competing VLA methods like UniDriveVLA.

Closed-Loop Evaluation (Bench2Drive)

  • VLGA achieved a state-of-the-art Driving Score of 79.08, outperforming the strongest prior VLA baseline (UniDriveVLA) by +0.71.
  • It demonstrated notable improvements in Success Rate (52.73%) and Comfortness (13.06) while maintaining comparable efficiency.
Qualitative trajectory planning comparison on nuScenes.

Limitations and Future Work

  • Inference Cost: Because VLGA relies on large vision-language backbones, it incurs a substantial computational footprint. Deploying it on edge hardware will require future work in model distillation and quantization tailored for autonomous driving.
  • Temporal Consistency: The current pointmap supervision is applied on a per-frame basis. The authors note that extending this supervision to enforce temporal consistency across multi-frame input windows is a promising direction to further sharpen long-horizon geometric reasoning.

Summary updated

Abstract

Vision-language-action (VLA) models can describe scenes and reason about them in language, yet still struggle to ground their actions in the dense 3D world around them. Existing approaches either inject features from a frozen 3D foundation model without an objective that ensures the policy uses them, or constrain geometry with sparse box and map losses that provide no dense spatial signal. We introduce VLGA, the first vision-language-action model supervised to reconstruct the dense 3D world it drives through. VLGA introduces geometry as a fourth modality alongside vision, language, and action through a dedicated expert supervised by a per-pixel pointmap regression loss against LiDAR. Extensive experiments conducted on challenging nuScenes and Bench2Drive datasets for open-loop and closed-loop evaluations, respectively, show the superiority of VLGA over counterpart VLA methods. In particular, on open-loop nuScenes, VLGA sets a new state of the art among VLA methods without ego status, with the lowest L2 (0.50\,m average) and 3-second collision rate (0.18\%). On closed-loop Bench2Drive, VLGA attains the state-of-the-art driving score of 79.08, +0.71 over the strongest prior VLA, at comparable efficiency and comfort.
BibTeX
@misc{yao2026vlga,
  author = {Jin Yao and Dhruva Dixith Kurra and Tom Lampo and Zezhou Cheng and Danhua Guo and Burhan Yaman},
  title = {{VLGA: Vision-Language-Geometry-Action Models for Autonomous Driving}},
  year = {2026},
  eprint = {2606.12396},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.12396},
}

Loading related content...

Figure