TL;DR
This paper introduces LAV, a novel end-to-end driving system that learns from the driving experiences of all vehicles in a scene, not just the ego-vehicle. By leveraging data from other agents, LAV significantly improves sample efficiency and robustness, achieving state-of-the-art performance on the CARLA benchmark.
What Problem Does This Paper Address?
Autonomous driving systems struggle with the "long tail" of driving scenarios – rare but critical situations that are difficult to encounter and collect data for, especially for a single ego-vehicle. Traditional methods rely on hand-designed rules or massive datasets, which are often insufficient for generalizing to the vast diversity of real-world driving. This paper tackles this challenge by proposing a system that learns from the collective experience of all observed vehicles, effectively creating a richer and more diverse training signal without requiring additional data collection.
Methods
LAV employs a three-stage modular pipeline:
- Perception Module: This module takes multi-modal sensor inputs (RGB cameras and LiDAR) and generates a viewpoint-invariant representation of the surrounding world. This is achieved through auxiliary supervision from 3D detection and semantic segmentation tasks, ensuring that the learned features are consistent across different vehicle perspectives.
- Motion Planner: This is the core innovation. Instead of just learning from the ego-vehicle's trajectory, LAV learns to predict future waypoints for all detected vehicles. It uses a two-stage approach: first, a geometric GPS target and discrete high-level command are used to predict coarse trajectories for all vehicles. Second, an iterative refinement module further refines the ego-vehicle's trajectory using its high-level command and GNSS goal.
- Controller: A low-level controller translates the refined motion plan into actual steering and acceleration commands. Crucially, this controller also considers the predicted motion plans of other vehicles to make emergency stop decisions, enhancing safety.
A key technique is privileged distillation, where a privileged motion planner (trained with access to ground truth perception and high-level commands) is used to supervise a motion planner that operates on inferred perception outputs. This allows the system to learn from richer, albeit simulated, ground truth while still functioning with real-world sensor inputs.
Datasets
The system is trained and evaluated using the CARLA simulator [17]. For the online leaderboard submission, the dataset consists of 400K frames collected across 8 publicly available towns with randomized weather conditions. For ablation studies, a subset of 4 towns was used, resulting in 186K frames, and tested on two unseen towns.
Key Insights and Ideas
- Learning from the collective: The most significant insight is that learning from the driving experiences of all vehicles in a scene provides a much richer and more diverse training signal than relying solely on the ego-vehicle. This directly addresses the long-tail problem.
- Viewpoint-invariant representation: The success of learning from other vehicles hinges on creating perception features that are independent of the observer's viewpoint. Auxiliary tasks like 3D detection and segmentation help achieve this.
- Privileged distillation: This technique is crucial for bridging the gap between simulated ground truth and real-world sensor inputs, enabling the system to leverage rich supervisory signals without requiring perfect real-world ground truth for all aspects.
- Vehicle-aware control: Incorporating the predicted future trajectories of other vehicles into the control loop is a vital step towards safer autonomous driving, allowing for proactive collision avoidance.
Key Results
LAV achieved state-of-the-art performance on the public CARLA Leaderboard, significantly outperforming prior methods:
- Driving Score: 61.85 (25 points higher than the previous best)
- Route Completion Rate: 94.46 (24 points higher than the previous best)
- The system also won the 2021 CARLA Autonomous Driving challenge.
- Ablation studies showed that training on other vehicles' trajectories and using privileged distillation were critical for achieving high performance.
Limitations and Future Work
- Simulation-to-real gap: The authors acknowledge that the system is trained and evaluated solely in simulation. Direct deployment in the real world would likely still result in traffic accidents, highlighting the ongoing challenge of sim-to-real transfer.
- Behavior prediction: The current behavior predictor, while effective, does not explicitly consider multi-modality beyond high-level commands. Extending this with probabilistic formulations could further improve its ability to handle diverse behaviors.
- Motion predictor generalization: The authors suggest improving the motion predictor beyond its current raster representation as an exciting future direction.
Summary updated