TL;DR
This paper introduces METDrive, a novel end-to-end autonomous driving system that enhances waypoint prediction by incorporating temporal guidance from the ego vehicle's own state (like steering and throttle) alongside traditional sensor data. This approach leads to more robust and safer driving decisions, outperforming existing methods on the CARLA benchmark.
What Problem Does This Paper Address?
End-to-end autonomous driving systems aim to directly map sensor inputs (like cameras and LiDAR) to driving actions. While multimodal sensor fusion has improved these systems, they often struggle with scenarios where irrelevant objects in the environment can mislead the model. Furthermore, existing methods tend to overlook the crucial information contained within the ego vehicle's own dynamic state over time. This can lead to unexpected failures, such as the ego vehicle stopping at a green light, even in clear conditions, as observed in the paper's experiments. The paper argues that incorporating the vehicle's own temporal state (e.g., steering angle, throttle, rotation) is essential for more informed and reliable motion planning.
Methods
METDrive employs a dual-branch architecture:
- Perception Branch: This branch processes image and LiDAR data using ResNet encoders to extract geometric features representing the environment's spatial layout. An attention-based fusion block is used to integrate these features effectively.
- Temporal Branch: This novel component treats the ego vehicle's states (rotation angles, steering, throttle, and waypoint vectors) as time-series data. These signals are decomposed into horizontal and vertical components and then encoded using positional and token embeddings, followed by self-attention encoders.
The features from both branches are then concatenated and fed into a Gated Recurrent Unit (GRU)-based waypoint decoder. A key innovation is the Temporal Guidance Loss, which encourages consistency between waypoint predictions made using features from different temporal proximities. This loss function helps the model learn to better align the temporal ego-state information with the geometric perception features.
Datasets
- Training Data: Collected in CARLA Towns 01, 03, 04, 06, 07, and 10, including front camera images, central LiDAR point clouds, rotation angle records, control signals, velocity records, control command records, and target point records. The data was filtered and averaged to mitigate noise from autonomous agent recordings.
- Evaluation Benchmark: CARLA Longest6 benchmark.
Key Insights and Ideas
- Ego-State is Crucial: The paper strongly emphasizes that a vehicle's own dynamic state over time is a vital, yet often overlooked, input for robust end-to-end driving. Ignoring it can lead to seemingly illogical failures.
- Temporal Guidance for Robustness: By explicitly modeling the temporal evolution of ego states and using a temporal guidance loss, METDrive can better integrate this information with perception, leading to more stable and context-aware decisions.
- Decomposition for Clarity: Decomposing temporal signals (like steering and rotation) into horizontal and vertical components helps clarify lateral and longitudinal features, improving the encoder's ability to process them.
Key Results
- CARLA Longest6 Benchmark Performance:
- Driving Score (DS): 70%
- Route Completion (RC): 94%
- Infraction Score (IS): 0.78
- METDrive outperformed several recent approaches on the CARLA Longest6 benchmark, achieving better DS, RC, and IS.
- Qualitative results showed that METDrive produced smoother speed outputs (ranging from 7 km/h to 22 km/h) compared to a baseline (TransFuser++), which exhibited unnatural oscillations between 0 km/h and 22 km/h, leading to jerky driving.
- An ablation study demonstrated that removing the temporal guidance loss significantly degraded METDrive's performance, highlighting its importance.
Limitations and Future Work
- Dataset Quality: The authors acknowledge that the dataset, generated by autonomous agents in CARLA, exhibits inconsistencies and lacks the smoothness of human driving. This noise in ego-state data required significant preprocessing.
- Future Work:
- Acquire a higher-fidelity dataset sourced from expert human drivers to reduce noise and improve data quality.
- Develop a more robust framework with a broader spectrum of time series encoders and feature fusion techniques tailored for ego-state inputs.
Summary updated