METDrive: Multi-modal End-to-end Autonomous Driving with Temporal Guidance

Ziang Guo, Xinhao Lin, Zakhar Yagudin, Artem Lykov, Yong Wang, Yanqiang Li, Dzmitry Tsetserukou

Abstract

Multi-modal end-to-end autonomous driving has shown promising advancements in recent work. By embedding more modalities into end-to-end networks, the system's understanding of both static and dynamic aspects of the driving environment is enhanced, thereby improving the safety of autonomous driving. In this paper, we introduce METDrive, an end-to-end system that leverages temporal guidance from the embedded time series features of ego states, including rotation angles, steering, throttle signals, and waypoint vectors. The geometric features derived from perception sensor data and the time series features of ego state data jointly guide the waypoint prediction with the proposed temporal guidance loss function. We evaluated METDrive on the CARLA leaderboard benchmarks, achieving a driving score of 70%, a route completion score of 94%, and an infraction score of 0.78.

Introduction

Most modern end-to-end autonomous driving models rely heavily on perception sensors like cameras and LiDAR to navigate. However, these models often suffer from a lack of context regarding the vehicle's own historical state. This deficiency frequently leads to a "freezing" phenomenon, where vehicles stop at green lights for no apparent reason or produce jerky, oscillating movements. Without a sense of continuity—or "temporal guidance"—systems can become confused by irrelevant environmental objects or fail to maintain necessary momentum.

Existing models often suffer from a lack of temporal context, leading to errors such as stopping at green lights.

Methodology

METDrive addresses these challenges through a dual-branch architecture designed to integrate spatial and temporal information.

Perception and Temporal Branches

The Perception Branch uses ResNet-based encoders to process RGB images and LiDAR point clouds, fusing them via an attention-based block to establish a geometric understanding of the environment.

The Temporal Branch serves as the core innovation. It processes a sequence of the vehicle's recent states—rotation angles, steering inputs, throttle signals, and waypoint vectors—as time-series data. These signals are decomposed into horizontal and vertical components relative to the vehicle's coordinate system and encoded using positional embeddings and self-attention mechanisms.

The METDrive architecture integrates multi-modal inputs, including perception sensors and historical ego-state time series.

Temporal Guidance Loss

To ensure the model effectively utilizes its history, the researchers designed a specific Temporal Guidance Loss function ($L_{Temporal}$). This function forces the model to maintain consistency between waypoint predictions made at different points in time. By employing importance sampling, the model learns to prioritize features closer to the current moment while respecting the overall intended trajectory.

Experimental Results

METDrive was trained on 320,000 samples across various CARLA simulator environments and evaluated on the challenging CARLA Longest6 benchmark. The system outperformed several high-profile models in both safety and efficiency.

Method Driving Score (DS) ↑ Route Completion (RC) ↑ Infraction Score (IS) ↑
METDrive (Ours) 70 94 0.78
TransFuser++ 69 94 0.72
ThinkTwice 67 77 0.84
DriveAdapter 59 82 0.68

Qualitatively, METDrive eliminated the unnatural speed oscillations seen in baseline models like TransFuser++. While baselines often fluctuated wildly between 0 and 22 km/h, METDrive maintained a smooth range (7 km/h to 22 km/h), resulting in a more human-like driving experience.

Conclusion and Future Work

By integrating ego-states as vital temporal guidance, METDrive significantly improves the stability and reliability of end-to-end autonomous driving. Future work will focus on sourcing high-quality training data from expert human drivers to teach the model more natural navigation nuances and refining the fusion of temporal and geometric features to handle complex urban scenarios.

Summary updated

Abstract

Multi-modal end-to-end autonomous driving has shown promising advancements in recent work. By embedding more modalities into end-to-end networks, the system's understanding of both static and dynamic aspects of the driving environment is enhanced, thereby improving the safety of autonomous driving. In this paper, we introduce METDrive, an end-to-end system that leverages temporal guidance from the embedded time series features of ego states, including rotation angles, steering, throttle signals, and waypoint vectors. The geometric features derived from perception sensor data and the time series features of ego state data jointly guide the waypoint prediction with the proposed temporal guidance loss function. We evaluated METDrive on the CARLA leaderboard benchmarks, achieving a driving score of 70%, a route completion score of 94%, and an infraction score of 0.78.
BibTeX
@misc{guo2024metdrive,
  author = {Ziang Guo and Xinhao Lin and Zakhar Yagudin and Artem Lykov and Yong Wang and Yanqiang Li and Dzmitry Tsetserukou},
  title = {{METDrive: Multi-modal End-to-end Autonomous Driving with Temporal Guidance}},
  year = {2024},
  eprint = {2409.12667},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.12667},
}
Figure