AutoE2E: Don't speak - Language free end-to-end autonomy A Real-Time Driving Model for L2++ ADAS and L4 Robotaxis

Muhammad Zain Khawaja, Zaynab El Mawas, Ryota Yamada, Gabriela Cordova, Arseni Lysak, Intisar Chowdhury, Bharat Jayaprakash, Apostolos Georgiadis, Shri Harsha Adapala, Jatin Gautam, Christoph Bank, Alejandra Solarte Uscategui, Atanasko Boris Mitrev, Hassan Ali Ansari, Nishad SahuarXiv 2026 · arXiv

TL;DR

AutoE2E is a high-efficiency, open-source reactive driving model that rejects the current trend of using large language models for "reasoning." By utilizing a compact temporal Bird’s-Eye-View (BEV) architecture with only 1.45M trainable parameters, it achieves real-time performance (up to 76 FPS) and competitive accuracy on the KITScenes benchmark.

What Problem Does This Paper Address?

The autonomous driving research community is currently enamored with Vision-Language-Action (VLA) models. These models produce "reasoning traces" (textual explanations of their actions), but they come with significant baggage: they are computationally massive, slow, and their linguistic explanations are often unverified—there is no guarantee the text actually reflects the underlying neural logic.

The Autoware Foundation Robotaxi Working Group argues for a different path. They address the need for a deployable, inspectable, and real-time end-to-end (E2E) model that fits within the inference budgets of actual vehicle hardware. They aim to provide a transparent baseline that proves you don't need a billion-parameter LLM to handle complex urban navigation.

Methods

AutoE2E is the "reactive band" of a planned three-band architecture (Reactive, Reasoning, and World-Action).

Figure 1: The working group’s planned three-band architecture.

Key technical features include:

  • Temporal BEV Backbone: It uses a frozen BEVFormer v2 (ResNet-50) backbone to process six camera views. It fuses eight frames of history (3.5 seconds) into a unified BEV representation.
  • Navigation Pathway: Unlike many camera-only models, AutoE2E explicitly consumes HD-map rasters (14 channels) and route-conditioning rasters (2 channels). These are fused into the camera BEV using deformable cross-attention, acting as a "stabilization mechanism" for the frozen vision features.
  • Deterministic Planning: A lightweight GRU-based recurrent planner predicts acceleration and curvature, which are then integrated via a unicycle model to produce a 6.4-second trajectory.
  • Parameter Efficiency: The model is remarkably lean. Out of 79.9M total parameters, 98% are frozen. Only 1.45M parameters are trained, focusing the learning entirely on the planning and navigation fusion tasks.
Figure 1: The working group’s planned three-band architecture. The reactive model at 10 Hz (violet band) is AutoE2E, the subject of this paper, already implemented and evaluated here. The world-action model at 1 Hz and the reasoning model at 1 Hz (grey bands) are planned interfaces, not yet implemented or trained; no result in this paper depends on them. The world-action band’s objective is explicitly non-linguistic: a JEPA-style future-feature reconstruction loss in the camera’s own embedding space, not a language-model reasoning trace – the design choice the paper’s title refers to.

Datasets

The authors utilize two primary datasets:

  1. nuPlan: Used for the initial large-scale trajectory and route training stage.
  2. KITScenes: A multimodal European dataset from the Karlsruhe Institute of Technology. This is the primary benchmark for fine-tuning and evaluation, providing high-fidelity camera data paired with Lanelet2 HD maps.
  3. EDGAR: Real-vehicle data from the Technical University of Munich used for preliminary on-vehicle validation.

Key Insights and Ideas

  • "Don't Speak" as a Design Choice: The title is a direct critique of VLA models. The authors argue that "inspectability" should come from open-source code and visualization tools (like their released DataModelConsole) rather than potentially hallucinated language traces.
  • The Power of Frozen Backbones: The paper demonstrates that a massive, pretrained vision backbone doesn't need to be unfrozen to learn driving policy. By training a tiny "navigation gate," they can effectively steer the existing vision features.
  • JEPA-style Future Prediction: Instead of predicting tokens, the roadmap for the "World-Action" band uses a Joint-Embedding Predictive Architecture (JEPA) to predict future latent features in the camera's own embedding space.
  • The "Copycat" Failure Mode: In closed-loop testing, they identified a common imitation learning trap: the model over-relies on its own ego-motion history. If the vehicle drifts slightly, the model "copies" that drift into its future plan. Zeroing out history at inference time was found to mitigate this.

Key Results

  • Accuracy: On the KITScenes validation set, the fine-tuned model achieved an Average Displacement Error (ADE) of 0.745 m and a Final Displacement Error (FDE) of 2.020 m at a 3-second horizon.
  • Efficiency: The model reaches up to 76 FPS on a consumer GPU (RTX 5080). On a workstation-grade A6000, latency is approximately 101 ms.
  • Benchmarking: In a cross-model comparison on KITScenes, AutoE2E outperformed or matched much larger models like Qwen-Drive-1.0 (4B parameters) and Alpamayo 1.5 (10B parameters) in open-loop FDE, despite being orders of magnitude smaller.

Limitations and Future Work

  • Open-Loop vs. Closed-Loop: The current quantitative success is strictly open-loop. In initial closed-loop simulations (AlpaSim), the model struggled, leaving the road in four out of five scenes. This highlights the "distribution shift" common in imitation learning.
  • HD-Map Dependency: The current "deployment contract" requires an HD map. Future iterations aim to replace this with standard navigation or SD maps to improve scalability.
  • Static Vision: The camera BEV pathway remains frozen. The authors plan to explore joint fine-tuning (unfreezing the backbone) to see if perception can be better optimized for specific driving maneuvers.
  • Data Diversity: The training is currently limited to a small number of German and multi-city scenes, leaving questions about performance in significantly different geographies or weather conditions.

Summary updated

Abstract

AutoE2E is an open-source end-to-end driving model developed by the Autoware Foundation Robotaxi Working Group. It introduces a temporal bird's-eye-view (BEV) architecture that combines multi-camera perception, HD-map information and route conditioning to predict future vehicle motion. The evaluated model uses a pretrained camera-BEV backbone, temporal feature fusion and a lightweight recurrent trajectory planner. It contains approximately 79.9 million parameters, of which approximately 1.45 million are trainable in the evaluated configuration. The v1.0 release provides model checkpoints, training and evaluation code, and DataModelConsole, a dashboard for visualizing and inspecting model experiments. On the KITScenes validation benchmark, the fine-tuned checkpoint achieves an average displacement error (ADE) of 0.745 m and a final displacement error (FDE) of 2.020 m at a three-second prediction horizon. The paper also reports preliminary real-vehicle research conducted with the Technical University of Munich's EDGAR platform, alongside ongoing collaboration with the Karlsruhe Institute of Technology. AutoE2E v1.0 is an early open research release. Its quantitative evaluation is open-loop, and the model has not yet demonstrated validated closed-loop autonomous driving. The release is intended to provide the autonomous-driving community with an inspectable baseline for further research, experimentation and collaborative development.
BibTeX
@misc{khawaja2026autoe,
  author = {Muhammad Zain Khawaja and Zaynab El Mawas and Ryota Yamada and Gabriela Cordova and Arseni Lysak and Intisar Chowdhury and Bharat Jayaprakash and Apostolos Georgiadis and Shri Harsha Adapala and Jatin Gautam and Christoph Bank and Alejandra Solarte Uscategui and Atanasko Boris Mitrev and Hassan Ali Ansari and Nishad Sahu},
  title = {{AutoE2E: Don't speak - Language free end-to-end autonomy A Real-Time Driving Model for L2++ ADAS and L4 Robotaxis}},
  year = {2026},
  eprint = {2611.22374},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22374},
}
Figure