TL;DR
VLGA introduces a dedicated, parameter-isolated geometry expert into Vision-Language-Action (VLA) models for autonomous driving, supervised by a dense per-pixel pointmap reconstruction loss against LiDAR. By combining language reasoning with dense 3D spatial grounding, VLGA achieves state-of-the-art performance on both open-loop (nuScenes) and closed-loop (Bench2Drive) autonomous driving benchmarks, significantly reducing long-horizon collision rates.
What Problem Does This Paper Address?
End-to-end Vision-Language-Action (VLA) models excel at high-level scene understanding and language reasoning, but they struggle with the continuous spatial precision required for safe trajectory planning. Driving is inherently a 3D spatial task, and language tokens alone cannot accurately capture dense geometric structures.
Existing approaches to ground VLA models in 3D space fall into three flawed paradigms:
- Sparse 3D Perception: They use bounding boxes or occupancy grids, which lack dense spatial grounding.
- Feature Injection: They inject dense 3D features directly into the Large Language Model (LLM) stream, which forces the same parameters to handle both text and geometry without a dedicated parameter budget or explicit geometric objective.
- Geometry-Only Models: They achieve dense spatial grounding but drop the language reasoning stream entirely.
VLGA bridges this gap by asking: Can we give dense 3D geometry its own dedicated parameter budget within a VLA policy while keeping the language stream completely intact?
Methods
The authors propose VLGA (Vision-Language-Geometry-Action), a four-expert driving policy built on a Mixture-of-Transformers (MoT) backbone.
The Four-Expert Architecture
VLGA processes multi-view camera inputs, navigation instructions, and ego status using four specialized experts coupled via masked joint attention:
- Understanding Expert (U): Handles language and scene semantics.
- Perception Expert (P): Processes sparse agent, map, and occupancy queries.
- Geometry Expert (G): Processes dense spatial structures using features from a pretrained, frozen geometry backbone (DVGT-2).
- Action Expert (A): Emits the final trajectory via flow matching, conditioning its predictions on the other three streams.
Dense Pointmap Supervision
To ensure the geometry stream actually encodes task-relevant 3D structures, the authors introduce a per-pixel pointmap regression loss against ground-truth LiDAR data during training. A lightweight transformer decoder predicts 3D points in the ego LiDAR frame alongside an uncertainty logit. Crucially, no LiDAR is required at inference time; this decoder is discarded after training.
Two-Stage Training Schedule
To prevent the geometric loss from interfering with the action expert's optimization trajectory, training is split into two stages:
- Geometry Stage: The action expert and inherited streams are frozen. Only the geometry expert, projector, and pointmap decoder are trained using the pointmap loss.
- Joint Stage: The action expert is unfrozen, and the model is trained on a combined loss of action (flow matching) and a reduced-weight pointmap loss.
Datasets
VLGA was trained and evaluated on two prominent autonomous driving benchmarks:
- nuScenes: Used for open-loop trajectory planning evaluation, featuring 6,019 validation samples from 150 urban driving scenes.
- Bench2Drive: A closed-loop driving benchmark containing 220 driving routes across 12 CARLA simulator towns, evaluating the model under active closed-loop control.
Key Insights and Ideas
- Dedicated Parameter Budgets Matter: Isolating the geometry expert within the MoT architecture allows the model to learn dense spatial representations without diluting the LLM's language capabilities.
- Supervision Over Mere Injection: Simply feeding 3D features into a model isn't enough. Explicitly forcing the geometry stream to reconstruct the 3D world via pointmap regression ensures the network retains critical spatial details.
- Safety-Critical Gains: The empirical benefits of dense geometry are concentrated on safety-critical, long-horizon metrics (like avoiding collisions 3 seconds out) and spatial-precision-demanding scenarios (like "Give Way" and "Emergency Braking"), rather than just improving average trajectory displacement (L2 error).
Key Results
Open-Loop Evaluation (nuScenes)
In the leakage-free "without ego status" setting, VLGA-Large outperformed all counterpart VLA methods on 15 out of 16 planning metrics:
- Achieved the lowest ST-P3 L2 average error of 0.50 m.
- Recorded a 3-second collision rate of 0.18%, which is roughly 2.5–3× lower than competing VLA methods like UniDriveVLA.
Closed-Loop Evaluation (Bench2Drive)
- VLGA achieved a state-of-the-art Driving Score of 79.08, outperforming the strongest prior VLA baseline (UniDriveVLA) by +0.71.
- It demonstrated notable improvements in Success Rate (52.73%) and Comfortness (13.06) while maintaining comparable efficiency.
Limitations and Future Work
- Inference Cost: Because VLGA relies on large vision-language backbones, it incurs a substantial computational footprint. Deploying it on edge hardware will require future work in model distillation and quantization tailored for autonomous driving.
- Temporal Consistency: The current pointmap supervision is applied on a per-frame basis. The authors note that extending this supervision to enforce temporal consistency across multi-frame input windows is a promising direction to further sharpen long-horizon geometric reasoning.
Summary updated