DRIFT: Detachable Reasoning for Runtime-Flexible and Efficient End-to-End Driving

Emad Aghajanzadeh, Bethany Anne Wong, Leonhard Kunze, Moritz Thoma, Maximilian Forstenhäusler, Nael Fasfous, Manoj Rohit Vemparala, Pierpaolo Mori, Christos Anagnostopoulos, Daniel Mueller-Gritschneder, Ulf SchlichtmannarXiv 2026 · arXiv

TL;DR

DRIFT is a new end-to-end autonomous driving framework that proves you don't need a massive 7-billion-parameter model for high-level reasoning. By using a compact language model (<150M parameters) and a "detachable" design, it achieves state-of-the-art driving performance while remaining efficient enough to run on edge hardware like the NVIDIA Jetson.

What Problem Does This Paper Address?

End-to-end (E2E) driving models—which map raw sensor data directly to steering and throttle—often act as "black boxes," lacking interpretability. To fix this, researchers have started integrating Vision-Language Models (VLMs) to provide scene descriptions and "reasoning" (e.g., "I am stopping because there is a pedestrian").

However, current VLM-based drivers face two major hurdles:

  1. Computational Cost: Most use massive models (7B+ parameters) that are too slow for real-time use on vehicle hardware.
  2. Rigid Architectures: Language reasoning is usually "baked in." You can't simply turn off the expensive language part to save power without breaking the entire driving pipeline.

The authors of DRIFT tackle these issues by creating a system that is both compact and detachable.

Methods

The DRIFT architecture consists of three main components:

  • The Backbone: A TransFuser-based perception system that extracts dense features from cameras and LiDAR.
  • The Compact LM: A tiny 135M-parameter language model (SmolLM2) that performs Visual Question Answering (VQA) and provides action commentary.
  • The Detachable Planner: A unified planning head that can be "conditioned" on two different sources: the actual Language Model (LM) or a lightweight surrogate reasoning module.
Figure 1. Overview of our proposed method architecture (left) and qualitative driving results (right).

The "secret sauce" is knowledge distillation. During training, the surrogate module learns to mimic the LM's internal reasoning tokens. At runtime, the system can bypass the LM entirely and use the surrogate tokens instead, or interleave them at a chosen "reasoning rate" (e.g., invoking the LM every 2 frames).

Datasets

The researchers evaluated DRIFT across three major benchmarks:

  • Bench2Drive: A challenging closed-loop benchmark in the CARLA simulator.
  • NAVSIM: A real-world dataset focused on non-reactive simulation.
  • AlpaSim: A real-world benchmark based on the nuPlan dataset.

Training was performed using the LEAD dataset, with VQA and commentary annotations generated by larger teacher models like DriveLM and Qwen3-VL.

Key Insights and Ideas

  • Small is Sufficient: You don't need a 7B model to understand a driving scene. By efficiently projecting driving-specific features into the language space, a 135M model performs just as well for driving tasks.
  • Language Training Enriches Features: Interestingly, even when the LM is turned off, the model drives better than a "language-free" version. This suggests that teaching a model to "talk" about what it sees forces the perception backbone to learn richer, more useful features.
  • Complementary Attention: Grad-CAM visualizations show that the LM and the surrogate module "look" at different things. The LM maintains a broad contextual awareness (watching oncoming traffic and crossings), while the surrogate focuses narrowly on the immediate driving path.
Grad-CAM maps showing the different attention patterns of the LM path vs. the surrogate path.

Key Results

  • State-of-the-Art Performance: On Bench2Drive, DRIFT achieved a Driving Score of 97.03, outperforming much larger 1B-parameter models like LinkVLA.
  • Efficiency: With only 255M total parameters, it is significantly more compact than existing VLM-based drivers.
  • Real-Time Ready: On an NVIDIA Jetson AGX Orin, DRIFT runs at ~28 FPS in its detached mode and ~6 FPS when the LM is fully active.
  • Optimal Reasoning Rate: The authors found that invoking the LM every 2 frames (RR=2) actually yielded better driving scores than using it every single frame, likely because it provides a stable "semantic refresh" without overriding the high-frequency surrogate planner.
The effect of different reasoning rates (RR) on Driving Score and Success Rate.

Limitations and Future Work

While DRIFT allows for runtime flexibility, the current version uses a fixed reasoning rate. The authors suggest that future work could involve an adaptive trigger—a system that automatically decides to "wake up" the expensive LM only when the scene becomes complex (e.g., a busy intersection) and relies on the cheap surrogate for simple highway cruising. Additionally, while the model performs well on real-world benchmarks, more extensive closed-loop testing in diverse real-world environments is needed to fully prove its "detachable" reliability.

Summary updated

Abstract

The integration of vision-language models (VLMs) into end-to-end (E2E) autonomous driving has demonstrated strong potential by providing scene description and reasoning capabilities. However, VLMs are computationally expensive, and switching language reasoning on and off at runtime without hurting closed-loop performance has not been demonstrated, which hinders practical adoption. To address these issues, we propose DRIFT, an E2E autonomous driving framework that integrates a lightweight language model (LM) for visual question answering and action commentary. By efficiently projecting intermediate driving representations into the language embedding space, we demonstrate that a compact LM (<150M parameters) can achieve strong scene description and action commentary. Furthermore, we introduce a runtime-flexible planning module that is trained on surrogate reasoning features distilled from the LM and can be conditioned on either the LM or the surrogate reasoning features at inference. This design enables bypassing the LM entirely when required by deployment constraints, without sacrificing driving performance. With only 255M parameters, DRIFT reaches up to a Driving Score (DS) of 97.03 and a Success Rate of 89.39% (mean over three runs) on Bench2Drive by interleaving its LM and surrogate reasoning at a chosen reasoning rate. This is on par with the strongest non-language models while being far more compact than existing VLM-based models. DRIFT further achieves competitive results on the real-world NAVSIM and AlpaSim benchmarks at significantly lower computational cost than existing VLM-based approaches. Project page: https://drift-e2e.github.io/
BibTeX
@misc{aghajanzadeh2026drift,
  author = {Emad Aghajanzadeh and Bethany Anne Wong and Leonhard Kunze and Moritz Thoma and Maximilian Forstenhäusler and Nael Fasfous and Manoj Rohit Vemparala and Pierpaolo Mori and Christos Anagnostopoulos and Daniel Mueller-Gritschneder and Ulf Schlichtmann},
  title = {{DRIFT: Detachable Reasoning for Runtime-Flexible and Efficient End-to-End Driving}},
  year = {2026},
  eprint = {2611.24719},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24719},
}
Figure