Abstract
Multi-modal end-to-end autonomous driving has shown promising advancements in recent work. By embedding more modalities into end-to-end networks, the system's understanding of both static and dynamic aspects of the driving environment is enhanced, thereby improving the safety of autonomous driving. In this paper, we introduce METDrive, an end-to-end system that leverages temporal guidance from the embedded time series features of ego states, including rotation angles, steering, throttle signals, and waypoint vectors. The geometric features derived from perception sensor data and the time series features of ego state data jointly guide the waypoint prediction with the proposed temporal guidance loss function. We evaluated METDrive on the CARLA leaderboard benchmarks, achieving a driving score of 70%, a route completion score of 94%, and an infraction score of 0.78.
Introduction
Most modern end-to-end autonomous driving models rely heavily on perception sensors like cameras and LiDAR to navigate. However, these models often suffer from a lack of context regarding the vehicle's own historical state. This deficiency frequently leads to a "freezing" phenomenon, where vehicles stop at green lights for no apparent reason or produce jerky, oscillating movements. Without a sense of continuity—or "temporal guidance"—systems can become confused by irrelevant environmental objects or fail to maintain necessary momentum.
Methodology
METDrive addresses these challenges through a dual-branch architecture designed to integrate spatial and temporal information.
Perception and Temporal Branches
The Perception Branch uses ResNet-based encoders to process RGB images and LiDAR point clouds, fusing them via an attention-based block to establish a geometric understanding of the environment.
The Temporal Branch serves as the core innovation. It processes a sequence of the vehicle's recent states—rotation angles, steering inputs, throttle signals, and waypoint vectors—as time-series data. These signals are decomposed into horizontal and vertical components relative to the vehicle's coordinate system and encoded using positional embeddings and self-attention mechanisms.
Temporal Guidance Loss
To ensure the model effectively utilizes its history, the researchers designed a specific Temporal Guidance Loss function ($L_{Temporal}$). This function forces the model to maintain consistency between waypoint predictions made at different points in time. By employing importance sampling, the model learns to prioritize features closer to the current moment while respecting the overall intended trajectory.
Experimental Results
METDrive was trained on 320,000 samples across various CARLA simulator environments and evaluated on the challenging CARLA Longest6 benchmark. The system outperformed several high-profile models in both safety and efficiency.
| Method | Driving Score (DS) ↑ | Route Completion (RC) ↑ | Infraction Score (IS) ↑ |
|---|---|---|---|
| METDrive (Ours) | 70 | 94 | 0.78 |
| TransFuser++ | 69 | 94 | 0.72 |
| ThinkTwice | 67 | 77 | 0.84 |
| DriveAdapter | 59 | 82 | 0.68 |
Qualitatively, METDrive eliminated the unnatural speed oscillations seen in baseline models like TransFuser++. While baselines often fluctuated wildly between 0 and 22 km/h, METDrive maintained a smooth range (7 km/h to 22 km/h), resulting in a more human-like driving experience.
Conclusion and Future Work
By integrating ego-states as vital temporal guidance, METDrive significantly improves the stability and reliability of end-to-end autonomous driving. Future work will focus on sourcing high-quality training data from expert human drivers to teach the model more natural navigation nuances and refining the fusion of temporal and geometric features to handle complex urban scenarios.
Summary updated