Abstract
Autonomous driving systems often struggle to bridge the gap between high-level semantic reasoning and low-level trajectory generation. While classic end-to-end (E2E) methods excel at numerical control, they lack causal understanding. Conversely, Vision-Language Models (VLMs) offer sophisticated reasoning but struggle with precise spatial outputs. We present ORION, a holistic E2E framework that utilizes a Large Language Model (LLM) to generate "planning tokens" which instruct a generative planner. This approach aligns semantic logic with physical action, achieving state-of-the-art performance on the Bench2Drive benchmark with a +14.28 point increase in Driving Score.
1. Introduction: The Semantic-Action Gap
Autonomous driving has traditionally been bifurcated into two paradigms: classic end-to-end (E2E) methods and Vision-Language Model (VLM)-based approaches. Classic E2E methods are proficient at mapping sensor inputs to numerical trajectories but often lack the causal reasoning necessary for complex urban environments—they can calculate where to go but often fail to understand why a specific action is required.
In contrast, VLMs possess remarkable reasoning capabilities but are notoriously poor at "math." Asking an LLM to output raw numerical coordinates for a trajectory often leads to suboptimal, jittery, or physically impossible paths. ORION (hOlistic E2E autonomous dRiving framework by vIsion-language instructed actiON generation) bridges this "semantic-action gap" by using an LLM to provide high-level instructions to a dedicated generative planner, ensuring that high-level logic and low-level control are optimized together.
2. The ORION Architecture
The ORION framework is structured around three core pillars that facilitate the flow from raw perception to refined action:
- Vision Space (QT-Former): This module handles the heavy lifting of perception. It aggregates multi-view image features and maintains a Long-term Memory Bank. This allows the model to "remember" previous states, such as a traffic light that was red a few seconds ago, providing critical temporal context for decision-making.
- Reasoning Space (LLM): A fine-tuned Vicuna LLM processes the compressed scene features from the QT-Former. Instead of outputting a text-based path, it performs Visual Question Answering (VQA) tasks—describing the scene and analyzing critical objects. Most importantly, it generates a special "planning token" that encapsulates its high-level decision.
- Action Space (Generative Planner): This is the bridge between thought and movement. ORION uses a Variational Autoencoder (VAE) or a Diffusion model to map the LLM's planning token into a distribution of multi-modal trajectories. By using a generative model, ORION can represent the natural uncertainty of human driving, accounting for multiple "correct" ways to navigate a scenario.
3. Training and Datasets
To effectively train the LLM component, the authors developed Chat-B2D, a massive VQA dataset containing 2.11 million pairs. This dataset was created using an automated pipeline powered by Qwen2VL to generate high-quality descriptions and reasoning for driving scenes.
For evaluation, the framework was tested on Bench2Drive, a rigorous closed-loop simulator based on CARLA that requires agents to interact with dynamic environments across 44 distinct scenarios. Additionally, standard open-loop evaluations were performed on the nuScenes dataset to compare against classic state-of-the-art (SOTA) methods.
4. Experimental Results
ORION set a new benchmark for performance on the Bench2Drive leaderboard, significantly outperforming previous methods. It achieved a Driving Score (DS) of 77.74, representing a massive +14.28 point jump over the previous best (DriveTransformer). Its Success Rate (SR) reached 54.62%, a nearly 20% improvement.
Key insights from the results include:
- The Planning Token Strategy: Treating the trajectory as a latent variable conditioned on a semantic token preserves the LLM's reasoning power while utilizing the generative model's numerical precision.
- Synergy of Tasks: Training on auxiliary VQA tasks (like scene description) actually improves planning performance, suggesting that understanding the "why" directly aids the "how."
- Causal Reasoning: ORION showed particular strength in scenarios like "Overtaking" and "Emergency Braking," where understanding the intent of other agents is critical.
5. Conclusion
ORION demonstrates that a holistic approach—aligning high-level reasoning with generative action planning—is a powerful paradigm for autonomous driving. While the computational complexity of large VLMs remains a hurdle for real-time edge deployment, the framework's success highlights the potential of VLM-instructed systems. Future work will focus on model compression and pruning to achieve the low-latency performance required for real-world deployment without sacrificing the framework's reasoning capabilities.
Summary updated