TL;DR
AutoE2E is a high-efficiency, open-source reactive driving model that rejects the current trend of using large language models for "reasoning." By utilizing a compact temporal Bird’s-Eye-View (BEV) architecture with only 1.45M trainable parameters, it achieves real-time performance (up to 76 FPS) and competitive accuracy on the KITScenes benchmark.
What Problem Does This Paper Address?
The autonomous driving research community is currently enamored with Vision-Language-Action (VLA) models. These models produce "reasoning traces" (textual explanations of their actions), but they come with significant baggage: they are computationally massive, slow, and their linguistic explanations are often unverified—there is no guarantee the text actually reflects the underlying neural logic.
The Autoware Foundation Robotaxi Working Group argues for a different path. They address the need for a deployable, inspectable, and real-time end-to-end (E2E) model that fits within the inference budgets of actual vehicle hardware. They aim to provide a transparent baseline that proves you don't need a billion-parameter LLM to handle complex urban navigation.
Methods
AutoE2E is the "reactive band" of a planned three-band architecture (Reactive, Reasoning, and World-Action).
Key technical features include:
- Temporal BEV Backbone: It uses a frozen BEVFormer v2 (ResNet-50) backbone to process six camera views. It fuses eight frames of history (3.5 seconds) into a unified BEV representation.
- Navigation Pathway: Unlike many camera-only models, AutoE2E explicitly consumes HD-map rasters (14 channels) and route-conditioning rasters (2 channels). These are fused into the camera BEV using deformable cross-attention, acting as a "stabilization mechanism" for the frozen vision features.
- Deterministic Planning: A lightweight GRU-based recurrent planner predicts acceleration and curvature, which are then integrated via a unicycle model to produce a 6.4-second trajectory.
- Parameter Efficiency: The model is remarkably lean. Out of 79.9M total parameters, 98% are frozen. Only 1.45M parameters are trained, focusing the learning entirely on the planning and navigation fusion tasks.
Datasets
The authors utilize two primary datasets:
- nuPlan: Used for the initial large-scale trajectory and route training stage.
- KITScenes: A multimodal European dataset from the Karlsruhe Institute of Technology. This is the primary benchmark for fine-tuning and evaluation, providing high-fidelity camera data paired with Lanelet2 HD maps.
- EDGAR: Real-vehicle data from the Technical University of Munich used for preliminary on-vehicle validation.
Key Insights and Ideas
- "Don't Speak" as a Design Choice: The title is a direct critique of VLA models. The authors argue that "inspectability" should come from open-source code and visualization tools (like their released
DataModelConsole) rather than potentially hallucinated language traces. - The Power of Frozen Backbones: The paper demonstrates that a massive, pretrained vision backbone doesn't need to be unfrozen to learn driving policy. By training a tiny "navigation gate," they can effectively steer the existing vision features.
- JEPA-style Future Prediction: Instead of predicting tokens, the roadmap for the "World-Action" band uses a Joint-Embedding Predictive Architecture (JEPA) to predict future latent features in the camera's own embedding space.
- The "Copycat" Failure Mode: In closed-loop testing, they identified a common imitation learning trap: the model over-relies on its own ego-motion history. If the vehicle drifts slightly, the model "copies" that drift into its future plan. Zeroing out history at inference time was found to mitigate this.
Key Results
- Accuracy: On the KITScenes validation set, the fine-tuned model achieved an Average Displacement Error (ADE) of 0.745 m and a Final Displacement Error (FDE) of 2.020 m at a 3-second horizon.
- Efficiency: The model reaches up to 76 FPS on a consumer GPU (RTX 5080). On a workstation-grade A6000, latency is approximately 101 ms.
- Benchmarking: In a cross-model comparison on KITScenes, AutoE2E outperformed or matched much larger models like Qwen-Drive-1.0 (4B parameters) and Alpamayo 1.5 (10B parameters) in open-loop FDE, despite being orders of magnitude smaller.
Limitations and Future Work
- Open-Loop vs. Closed-Loop: The current quantitative success is strictly open-loop. In initial closed-loop simulations (AlpaSim), the model struggled, leaving the road in four out of five scenes. This highlights the "distribution shift" common in imitation learning.
- HD-Map Dependency: The current "deployment contract" requires an HD map. Future iterations aim to replace this with standard navigation or SD maps to improve scalability.
- Static Vision: The camera BEV pathway remains frozen. The authors plan to explore joint fine-tuning (unfreezing the backbone) to see if perception can be better optimized for specific driving maneuvers.
- Data Diversity: The training is currently limited to a small number of German and multi-city scenes, leaving questions about performance in significantly different geographies or weather conditions.
Summary updated