Abstract
Training autonomous ground vehicles typically necessitates extensive ground-level datasets, which are costly and logistically challenging to acquire. This paper presents a framework that leverages abundant aerial imagery to train driving policies and localization models. By utilizing Neural Radiance Fields (NeRF) as a bridge, the system synthesizes realistic ground-level perspectives from bird's-eye view (BEV) drone data. Our results demonstrate that models trained exclusively on these synthesized views can effectively navigate real-world environments, often outperforming traditional imitation learning methods trained on human-driven ground data.
Introduction
The development of autonomous vehicles (AVs) is currently bottlenecked by the requirement for massive amounts of ground-level sensor data. Collecting thousands of hours of video from street-level cameras is labor-intensive and restricted to accessible public roads. In contrast, aerial imagery from drones or satellites offers a scalable alternative, covering vast areas with minimal effort. However, a significant "viewpoint gap" exists between the high-altitude perspective of a drone and the bumper-height view of a robot car. This research bridges that gap by transforming aerial data into actionable ground-level training signals.
Methodology
The proposed "real-to-sim-to-real" framework centers on Neural Radiance Fields (NeRF) to reconstruct the environment and generate synthetic training data.
1. Reconstruction with Road Priors
The process begins with a sequence of aerial images and their associated poses. A NeRF model is trained to represent the 3D scene. To mitigate issues like occlusions from trees or buildings—common in aerial views—the authors incorporate "road priors." These priors provide the model with predefined information regarding road geometry and color, ensuring that synthesized ground-level views remain accurate and free of artifacts.
2. Data Generation and Task Learning
Once the NeRF is optimized, it acts as a high-fidelity simulator. Thousands of synthetic ground-level images are rendered along various driving trajectories. This data is then used for two primary tasks:
- Visuo-motor Policy Training: A convolutional neural network (CNN) is trained to map raw pixels to steering commands, using a pure-pursuit controller to provide ground-truth labels for the synthetic paths.
- Visual Localization: The model learns to estimate the vehicle's coordinates and heading (yaw) from a single ground-level image.
Experimental Setup
The framework was evaluated in a 1/10th scale "mini-city" environment. Aerial data consisted of approximately 100 images captured by a Parrot Bebop 2 drone. For evaluation, an Optitrack motion capture system provided ground-truth positioning. The deployment platform was a custom racecar equipped with an NVIDIA Jetson Xavier NX and a ZED 2i stereo camera.
Results
The aerial-trained models were compared against baselines trained on real human-driven ground data.
- Driving Performance: In end-to-end control tests, the proposed method significantly reduced the need for human intervention. While a standard imitation learning baseline required 10 interventions to complete a lap, the aerial-trained model required only one.
- Localization Accuracy: When fused with an Inertial Measurement Unit (IMU), the system achieved a position error of 0.09m to 0.12m and a heading error of 1.12° to 2.26°. Notably, the synthetic data often led to higher accuracy than real ground-view imagery, likely due to the reduced noise and increased consistency of NeRF-generated views.
Conclusion
This work demonstrates that NeRFs can serve as powerful, compact simulators for bridging the gap between aerial observation and ground-level action. By generating "off-nominal" data—such as views from incorrect lanes—the system teaches robots to recover from errors more effectively than traditional methods. Future research will focus on extending this framework to dynamic environments with moving actors and scaling to full-sized urban environments using satellite imagery.
Summary updated