METDrive: Multimodal End-to-End Autonomous Driving with Temporal Guidance

Ziang Guo, Xie Lin, Zakhar Yagudin, Artem Lykov, Yong Wang, Yong–Lu Li, Dzmitry Tsetserukou

TL;DR

This paper introduces METDrive, a novel end-to-end autonomous driving system that enhances waypoint prediction by incorporating temporal guidance from the ego vehicle's own state (like steering and throttle) alongside traditional sensor data. This approach leads to more robust and safer driving decisions, outperforming existing methods on the CARLA benchmark.

What Problem Does This Paper Address?

End-to-end autonomous driving systems aim to directly map sensor inputs (like cameras and LiDAR) to driving actions. While multimodal sensor fusion has improved these systems, they often struggle with scenarios where irrelevant objects in the environment can mislead the model. Furthermore, existing methods tend to overlook the crucial information contained within the ego vehicle's own dynamic state over time. This can lead to unexpected failures, such as the ego vehicle stopping at a green light, even in clear conditions, as observed in the paper's experiments. The paper argues that incorporating the vehicle's own temporal state (e.g., steering angle, throttle, rotation) is essential for more informed and reliable motion planning.

Methods

METDrive employs a dual-branch architecture:

  1. Perception Branch: This branch processes image and LiDAR data using ResNet encoders to extract geometric features representing the environment's spatial layout. An attention-based fusion block is used to integrate these features effectively.
  2. Temporal Branch: This novel component treats the ego vehicle's states (rotation angles, steering, throttle, and waypoint vectors) as time-series data. These signals are decomposed into horizontal and vertical components and then encoded using positional and token embeddings, followed by self-attention encoders.

The features from both branches are then concatenated and fed into a Gated Recurrent Unit (GRU)-based waypoint decoder. A key innovation is the Temporal Guidance Loss, which encourages consistency between waypoint predictions made using features from different temporal proximities. This loss function helps the model learn to better align the temporal ego-state information with the geometric perception features.

Datasets

  • Training Data: Collected in CARLA Towns 01, 03, 04, 06, 07, and 10, including front camera images, central LiDAR point clouds, rotation angle records, control signals, velocity records, control command records, and target point records. The data was filtered and averaged to mitigate noise from autonomous agent recordings.
  • Evaluation Benchmark: CARLA Longest6 benchmark.

Key Insights and Ideas

  • Ego-State is Crucial: The paper strongly emphasizes that a vehicle's own dynamic state over time is a vital, yet often overlooked, input for robust end-to-end driving. Ignoring it can lead to seemingly illogical failures.
  • Temporal Guidance for Robustness: By explicitly modeling the temporal evolution of ego states and using a temporal guidance loss, METDrive can better integrate this information with perception, leading to more stable and context-aware decisions.
  • Decomposition for Clarity: Decomposing temporal signals (like steering and rotation) into horizontal and vertical components helps clarify lateral and longitudinal features, improving the encoder's ability to process them.

Key Results

  • CARLA Longest6 Benchmark Performance:
    • Driving Score (DS): 70%
    • Route Completion (RC): 94%
    • Infraction Score (IS): 0.78
  • METDrive outperformed several recent approaches on the CARLA Longest6 benchmark, achieving better DS, RC, and IS.
  • Qualitative results showed that METDrive produced smoother speed outputs (ranging from 7 km/h to 22 km/h) compared to a baseline (TransFuser++), which exhibited unnatural oscillations between 0 km/h and 22 km/h, leading to jerky driving.
  • An ablation study demonstrated that removing the temporal guidance loss significantly degraded METDrive's performance, highlighting its importance.

Limitations and Future Work

  • Dataset Quality: The authors acknowledge that the dataset, generated by autonomous agents in CARLA, exhibits inconsistencies and lacks the smoothness of human driving. This noise in ego-state data required significant preprocessing.
  • Future Work:
    • Acquire a higher-fidelity dataset sourced from expert human drivers to reduce noise and improve data quality.
  • Develop a more robust framework with a broader spectrum of time series encoders and feature fusion techniques tailored for ego-state inputs.

Summary updated

Abstract

Multimodal end-to-end autonomous driving has shown promising advancements in recent work. By embedding more modalities into end-to-end networks, the system's understanding of both static and dynamic aspects of the driving environment is enhanced, thereby improving the safety of autonomous driving. In this paper, we introduce METDrive, an end-to-end system that leverages temporal guidance from the embedded time series features of ego states, including rotation angles, steering, throttle signals, and waypoint vectors. The geometric features derived from the perception sensor data and the time series features of ego state data jointly guide the waypoint prediction with the proposed temporal guidance loss function. We evaluated METDrive on the CARLA leaderboard benchmarks, achieving a driving score of 70%, a route completion score of 94%, and an infraction score of 0.78.
BibTeX
@misc{guo2024metdrive,
  author = {Ziang Guo and Xie Lin and Zakhar Yagudin and Artem Lykov and Yong Wang and Yong–Lu Li and Dzmitry Tsetserukou},
  title = {{METDrive: Multimodal End-to-End Autonomous Driving with Temporal Guidance}},
  year = {2024},
  eprint = {2409.12667},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.12667},
}
Figure