Abstract
Modern autonomous vehicle (AV) motion planners increasingly rely on imitation learning to navigate complex urban environments. However, the high cost of high-fidelity (HD) sensor suites—typically comprising LiDAR, RADAR, and high-definition cameras—limits the scale of data collection. This paper investigates whether large volumes of low-cost "commodity" sensor data can compensate for lower sensor quality. Our findings demonstrate that a motion planner trained on 100 hours of commodity stereo camera data, supplemented by a small amount of fine-tuning, outperforms a planner trained on 25 hours of expensive HD sensor data. This suggests a scalable path forward for addressing the "long tail" of driving scenarios through mass-market vehicle data collection.
1. Introduction
The transition toward imitation learning in AV motion planning has created a massive demand for real-world driving demonstrations. To safely navigate rare and complex scenarios, models require exposure to a vast diversity of data. Traditionally, this data is collected using specialized fleets equipped with expensive HD sensor suites. The prohibitive cost of these sensors makes it financially unfeasible to deploy them at the scale necessary to capture the "long tail" of driving events.
This study explores an alternative: using commodity-grade sensors, such as basic stereo cameras, to collect data at scale. While these sensors introduce significant noise and lower fidelity compared to HD suites, we hypothesize that the sheer volume of data can mitigate these technical shortcomings. We specifically address the challenge of the "domain gap"—the difference in perception noise between sensor types—and propose a fine-tuning strategy to align models across different hardware configurations.
2. Methodology
2.1 Mid-to-Mid Architecture
Rather than employing an end-to-end approach that maps raw pixels to control commands, we utilize a "mid-to-mid" stack. The system first processes raw sensor data into a 3D vectorized representation of the environment. This representation includes:
- Dynamic Agents: 3D bounding boxes for vehicles and pedestrians.
- Environmental State: Traffic light statuses and semantic map data (e.g., lane boundaries, crosswalks).
2.2 Planner Architecture
The motion planner is a transformer-based network trained via behavior cloning with perturbations. The model is designed to ingest the current environment state and a 3-second history of the scene to predict a 1.2-second future trajectory for the ego-vehicle.
2.3 Bridging the Domain Gap
A central challenge in using commodity data is the discrepancy in noise characteristics compared to HD sensors. To address this, we employ a fine-tuning strategy. The model is initially trained on a large corpus of commodity vision data. Subsequently, it undergoes a limited number of training epochs on a small subset of HD data. This process "aligns" the model, allowing it to interpret the cleaner HD signals while benefiting from the diverse scenarios learned from the larger, noisier dataset.
3. Experimental Setup
3.1 Dataset
The study utilizes a proprietary dataset consisting of 140 hours of driving snippets (10–25 seconds each) recorded in downtown San Francisco. The data is partitioned as follows:
- Training: 100 hours
- Fine-tuning: 10 hours
- Testing: 30 hours
Unique to this study, the collection vehicles were equipped with both HD and commodity sensor suites simultaneously. This dual-sensor setup allows for a direct ablative analysis, comparing how the exact same driving events are perceived and acted upon across different hardware configurations.
4. Results and Discussion
4.1 Performance Metrics
The primary metric for evaluation is Interventions per 1,000 miles (I1k) within a closed-loop simulation. A lower I1k indicates a more robust and reliable planner.
| Training Configuration | I1k (Lower is Better) |
|---|---|
| 25h HD Data | ~1,304 |
| 100h Commodity Vision (with fine-tuning) | ~912 |
| 100h HD Data (Gold Standard) | ~675 |
4.2 Quantity vs. Quality
The results confirm that quantity can indeed compensate for quality. Increasing the data volume by a factor of four (from 25h to 100h) allowed the commodity vision-based planner to significantly outperform the 25h HD-based planner. While the 100h HD model remains the gold standard, the cost-to-performance ratio of the commodity data suggests it is a more viable path for massive scaling.
4.3 Perception and Field of View
Analysis of the failures revealed that the commodity vision system's limited field of view (160° vs. 360° for HD) and shorter range (~60m vs. 100m+) contributed to specific types of errors, such as rear-end collisions in simulation. Despite these hardware limitations, the model learned to drive more cautiously in forward-facing scenarios, effectively leveraging the increased data to navigate complex urban interactions.
5. Conclusion and Future Work
This research demonstrates that large-scale, low-cost data collection is a technically sound strategy for improving AV motion planners. By using fine-tuning to bridge the domain gap, researchers can utilize data from consumer-grade vehicles to train models that rival those trained on much smaller, more expensive datasets.
Future work will focus on expanding the data scale to over 10,000 hours and exploring even more accessible sensor configurations, such as monocular camera setups. Additionally, moving beyond log-based simulations to reactive multi-agent simulations will provide a more nuanced understanding of how these planners perform in dynamic real-world environments.
Summary updated