AD-EE: Early Exiting for Fast and Reliable Vision-Language Models in Autonomous Driving

Nan GuanarXiv 2025 · DOI · arXiv

TL;DR

Autonomous driving systems are increasingly using powerful Vision-Language Models (VLMs), but their high latency hinders real-time performance. This paper introduces AD-EE, an "Early Exit" framework that intelligently determines when a VLM can stop processing layers without sacrificing accuracy, significantly reducing inference time and improving object detection.

What Problem Does This Paper Address?

Autonomous driving relies on sophisticated perception and decision-making systems. Vision-Language Models (VLMs) are emerging as a powerful tool to enhance these systems by bridging visual understanding with linguistic reasoning. However, VLMs, especially large ones, are computationally intensive and suffer from high inference latency. This latency is a critical bottleneck for real-time applications like autonomous driving, where split-second decisions are crucial for safety. A key reason for this latency is "over-inference," where VLMs continue processing layers even after they have reached a confident prediction. Existing methods for speeding up VLMs often focus on model compression or architecture changes, but this paper tackles the problem of over-inference directly by enabling VLMs to exit early.

Methods

The core of AD-EE is an "Early Exit" mechanism. Instead of always running through all layers of a VLM, AD-EE dynamically determines an optimal "exit layer." This is achieved through a novel three-stage process inspired by causal inference:

  1. Clean Run: The VLM runs through all layers to establish a baseline accuracy.
  2. Corrupted Run: Layers are progressively removed from the end of the VLM. If accuracy drops significantly, it suggests the removed layers were crucial. If accuracy remains high or even improves, it indicates a potential early exit point.
  3. Restoration and Explore Run: This stage refines the identified exit layer. It restores layers to ensure the early exit doesn't negatively impact accuracy and then explores further by removing layers from the beginning to find an even more optimal exit.

This causal inference approach aims to identify layers that are causally responsible for accurate predictions, rather than just layers that are correlated with them. The paper also leverages prompt engineering to guide VLMs towards making confident predictions earlier.

Datasets

The paper evaluates AD-EE on two large-scale real-world autonomous driving datasets:

  • WAYMO: A standard benchmark for autonomous driving perception, used here with 14,000 front-camera images.
  • CODA: A dataset specifically designed to focus on critical "corner cases" in autonomous driving, allowing for a more rigorous evaluation of VLM robustness.

Additionally, the method was tested on a real vehicle running the Autoware Universe platform.

Key Insights and Ideas

  • Over-inference is a significant bottleneck: VLMs often continue processing long after they've made a confident decision, wasting computational resources and increasing latency.
  • Early Exit is a viable solution: By intelligently stopping inference at an optimal layer, latency can be drastically reduced without compromising accuracy.
  • Causal inference provides a principled way to find optimal exit layers: Moving beyond simple heuristics, this approach helps understand the causal contribution of each layer to the final prediction.
  • Prompt engineering is crucial for early exit: Carefully designed prompts can guide VLMs to reach confident predictions earlier, making early exit more effective.
  • Larger models benefit more from early exit: While all models see improvements, larger VLMs with deeper architectures show more substantial gains in both accuracy and latency reduction with AD-EE.

Key Results

AD-EE demonstrated significant improvements across various VLMs and datasets:

  • Latency Reduction: Maximum latency reductions of up to 57.58% were achieved, particularly with the LLaVA-7B model.
  • Accuracy Improvement: Object detection accuracy saw maximum gains of up to 44%.
  • CODA Dataset: AD-EE enabled LLaVA-7B to achieve 100% accuracy on categories like Traffic Facility, Obstruction, and Misc, where full-layer inference failed entirely.
  • Real-World Performance: On a real vehicle, AD-EE improved accuracy by 22% and reduced latency by 17% for the DeepSeek-VL2-Tiny model.

Limitations and Future Work

  • Limited Real-World VLM Variety: Due to hardware constraints, only DeepSeek-VL2-Tiny was comprehensively tested on the real vehicle. More extensive testing with various VLMs in real-world scenarios is needed.
  • Prompt Diversity: While several prompt designs were tested, the exploration of prompt diversity remains incomplete. Future work could investigate techniques like Chain-of-Thought prompting to further unlock VLM potential for early exit.
  • Broader Task Evaluation: The primary evaluation focused on 2D object detection. Future work could explore AD-EE's effectiveness on other critical autonomous driving tasks like planning and decision-making.

Summary updated

Abstract

With the rapid advancement of autonomous driving, deploying Vision-Language Models (VLMs) to enhance perception and decision-making has become increasingly common. However, the real-time application of VLMs is hindered by high latency and computational overhead, limiting their effectiveness in time-critical driving scenarios. This challenge is particularly evident when VLMs exhibit over-inference, continuing to process unnecessary layers even after confident predictions have been reached. To address this inefficiency, we propose AD-EE, an Early Exit framework that incorporates domain characteristics of autonomous driving and leverages causal inference to identify optimal exit layers. We evaluate our method on large-scale real-world autonomous driving datasets, including Waymo and the corner-case-focused CODA, as well as on a real vehicle running the Autoware Universe platform. Extensive experiments across multiple VLMs show that our method significantly reduces latency, with maximum improvements reaching up to 57.58%, and enhances object detection accuracy, with maximum gains of up to 44%.
BibTeX
@misc{guan2025ad,
  author = {Nan Guan},
  title = {{AD-EE: Early Exiting for Fast and Reliable Vision-Language Models in Autonomous Driving}},
  year = {2025},
  eprint = {2506.05404},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.05404},
}
Figure