TL;DR
DRIFT is a new end-to-end autonomous driving framework that proves you don't need a massive 7-billion-parameter model for high-level reasoning. By using a compact language model (<150M parameters) and a "detachable" design, it achieves state-of-the-art driving performance while remaining efficient enough to run on edge hardware like the NVIDIA Jetson.
What Problem Does This Paper Address?
End-to-end (E2E) driving models—which map raw sensor data directly to steering and throttle—often act as "black boxes," lacking interpretability. To fix this, researchers have started integrating Vision-Language Models (VLMs) to provide scene descriptions and "reasoning" (e.g., "I am stopping because there is a pedestrian").
However, current VLM-based drivers face two major hurdles:
- Computational Cost: Most use massive models (7B+ parameters) that are too slow for real-time use on vehicle hardware.
- Rigid Architectures: Language reasoning is usually "baked in." You can't simply turn off the expensive language part to save power without breaking the entire driving pipeline.
The authors of DRIFT tackle these issues by creating a system that is both compact and detachable.
Methods
The DRIFT architecture consists of three main components:
- The Backbone: A TransFuser-based perception system that extracts dense features from cameras and LiDAR.
- The Compact LM: A tiny 135M-parameter language model (SmolLM2) that performs Visual Question Answering (VQA) and provides action commentary.
- The Detachable Planner: A unified planning head that can be "conditioned" on two different sources: the actual Language Model (LM) or a lightweight surrogate reasoning module.
The "secret sauce" is knowledge distillation. During training, the surrogate module learns to mimic the LM's internal reasoning tokens. At runtime, the system can bypass the LM entirely and use the surrogate tokens instead, or interleave them at a chosen "reasoning rate" (e.g., invoking the LM every 2 frames).
Datasets
The researchers evaluated DRIFT across three major benchmarks:
- Bench2Drive: A challenging closed-loop benchmark in the CARLA simulator.
- NAVSIM: A real-world dataset focused on non-reactive simulation.
- AlpaSim: A real-world benchmark based on the nuPlan dataset.
Training was performed using the LEAD dataset, with VQA and commentary annotations generated by larger teacher models like DriveLM and Qwen3-VL.
Key Insights and Ideas
- Small is Sufficient: You don't need a 7B model to understand a driving scene. By efficiently projecting driving-specific features into the language space, a 135M model performs just as well for driving tasks.
- Language Training Enriches Features: Interestingly, even when the LM is turned off, the model drives better than a "language-free" version. This suggests that teaching a model to "talk" about what it sees forces the perception backbone to learn richer, more useful features.
- Complementary Attention: Grad-CAM visualizations show that the LM and the surrogate module "look" at different things. The LM maintains a broad contextual awareness (watching oncoming traffic and crossings), while the surrogate focuses narrowly on the immediate driving path.
Key Results
- State-of-the-Art Performance: On Bench2Drive, DRIFT achieved a Driving Score of 97.03, outperforming much larger 1B-parameter models like LinkVLA.
- Efficiency: With only 255M total parameters, it is significantly more compact than existing VLM-based drivers.
- Real-Time Ready: On an NVIDIA Jetson AGX Orin, DRIFT runs at ~28 FPS in its detached mode and ~6 FPS when the LM is fully active.
- Optimal Reasoning Rate: The authors found that invoking the LM every 2 frames (RR=2) actually yielded better driving scores than using it every single frame, likely because it provides a stable "semantic refresh" without overriding the high-frequency surrogate planner.
Limitations and Future Work
While DRIFT allows for runtime flexibility, the current version uses a fixed reasoning rate. The authors suggest that future work could involve an adaptive trigger—a system that automatically decides to "wake up" the expensive LM only when the scene becomes complex (e.g., a busy intersection) and relies on the cheap surrogate for simple highway cruising. Additionally, while the model performs well on real-world benchmarks, more extensive closed-loop testing in diverse real-world environments is needed to fully prove its "detachable" reliability.
Summary updated