TL;DR
GridDriver is an efficient, Bird’s-Eye-View (BEV)-conditioned Large Language Model (LLM) framework designed for real-time autonomous driving. By replacing traditional compressed visual queries with spatially structured "grid tokens," it achieves competitive planning and reasoning performance while maintaining a 32.7 ms latency suitable for on-board deployment in research vehicles.
What Problem Does This Paper Address?
While Large Language Models (LLMs) have shown immense promise in "reasoning" about driving—explaining why a car should slow down or answering questions about a scene—bringing them on-board a real vehicle is a massive challenge.
Most current Vision-Language-Action (VLA) models suffer from two main issues:
- Efficiency: Autoregressive language generation is slow. If a planner has to wait for a full sentence to be generated before it can output a trajectory, the car will crash.
- Spatial Grounding: Many models use front-view images or highly compressed visual features (like Q-Formers). These often lose the precise geometric information—like the exact boundary of a drivable lane—required for safe trajectory planning.
GridDriver tackles these by combining a lightweight perception stack with a novel tokenization strategy and an asynchronous scheduling system that decouples fast planning from slower language generation.
Methods
The GridDriver architecture is split into two main stages: Perception and Planning & Reasoning.
- Perception: It uses BEVFormer-tiny to fuse 6-camera images into a unified Bird’s-Eye-View (BEV) representation. To keep things efficient, they use a ResNet-50 backbone and only three encoder layers.
- Grid Tokenizer: Instead of using a Q-Former to compress the scene into a few dozen abstract queries, GridDriver uses a "grid tokenizer." It divides the BEV map into non-overlapping 3x3 patches, which are flattened into 324 spatial tokens. This preserves the 2D layout of the scene, making it easier for the LLM to "see" where objects are.
- The LLM Core: The model uses Qwen3-1.7B, a small but capable LLM, fine-tuned with LoRA. It takes navigation instructions, the spatial tokens, and an optional ego-state as input.
- Multi-Head Output: The LLM's hidden state is passed to several heads:
- A Waypoint MLP for trajectory prediction.
- An Indicator Head for driving status icons (e.g., "yielding").
- The standard LM Head for natural language explanations.
- Asynchronous Scheduling: To enable real-time use, the system interleaves planning and reasoning. It produces a new trajectory every frame (10Hz) but generates the natural language explanation token-by-token across multiple frames, ensuring the "thinking" doesn't block the "driving."
Datasets
The authors evaluated GridDriver on several major benchmarks and their own real-world data:
- nuScenes: Used for open-loop planning and driving explanations.
- NAVSIM (v1.1 and v2): A modern benchmark for end-to-end driving that focuses on non-reactive simulation.
- nuScenes-QA: A dataset of 83,337 questions to test the model's scene-understanding capabilities.
- In-house Urban Dataset: 251 km of driving data recorded in a city with six cameras, LiDAR, and high-precision GNSS/INS, used for real-world validation on a research vehicle.
Key Insights and Ideas
- Spatial Structure Matters: The authors found that grid tokens retain significantly more "linearly decodable" information than Q-Formers. Linear probes showed that grid tokens were much better at identifying vehicle occupancy and drivable areas, which translated directly to lower planning errors.
- Language as a Steering Wheel: One of the most fascinating findings is how much the text input influences the physical path. By injecting "counterfactual" reasoning—like telling the model "A stopped car is 4m ahead" even when the road is clear—the researchers could force the model to shorten its planned trajectory.
- Efficiency through Optimization: By using FP16 precision and CUDA graphs, the team reduced LLM planning latency from 56 ms to 32.7 ms, proving that LLM-based planners can run comfortably at 30Hz on modern hardware (like an RTX 4090).
Key Results
- NAVSIM: Achieved 87.9% PDMS, placing it among the top-performing VLA models without requiring complex reinforcement learning or massive pre-training.
- nuScenes-QA: Reached 57.6% accuracy, demonstrating that the same model used for planning can also reason about the scene.
- On-Board Performance: In a real-world ROS 2 pipeline, the system achieved 9.4 plans per second, providing smooth, real-time trajectory overlays and driving explanations on a research vehicle.
- Ablation: Replacing the grid tokenizer with a Q-Former increased planning error (L2) from 0.47m to 0.50m and significantly slowed down inference.
Limitations and Future Work
Despite its success, GridDriver has clear hurdles to clear before it's "production-ready":
- Coarse Resolution: The BEV resolution is 2.048 meters per cell. This is quite coarse for fine-grained maneuvers like navigating tight construction zones.
- Language Dominance: The model sometimes trusts the text instructions too much, potentially ignoring visual evidence if the navigation command is contradictory. Strengthening visual grounding is a priority.
- Open-Loop Constraints: The current evaluation is largely open-loop. The authors note that future work must involve closed-loop validation (where the model's actions affect the future state) and reinforcement learning to improve robustness in diverse driving scenarios.
Summary updated