Learning autonomous driving from aerial imagery

Varun Murali, Guy Rosman, Sertac Karaman, Daniela Rus

Abstract

Training autonomous ground vehicles typically necessitates extensive ground-level datasets, which are costly and logistically challenging to acquire. This paper presents a framework that leverages abundant aerial imagery to train driving policies and localization models. By utilizing Neural Radiance Fields (NeRF) as a bridge, the system synthesizes realistic ground-level perspectives from bird's-eye view (BEV) drone data. Our results demonstrate that models trained exclusively on these synthesized views can effectively navigate real-world environments, often outperforming traditional imitation learning methods trained on human-driven ground data.

Introduction

The development of autonomous vehicles (AVs) is currently bottlenecked by the requirement for massive amounts of ground-level sensor data. Collecting thousands of hours of video from street-level cameras is labor-intensive and restricted to accessible public roads. In contrast, aerial imagery from drones or satellites offers a scalable alternative, covering vast areas with minimal effort. However, a significant "viewpoint gap" exists between the high-altitude perspective of a drone and the bumper-height view of a robot car. This research bridges that gap by transforming aerial data into actionable ground-level training signals.

Methodology

The proposed "real-to-sim-to-real" framework centers on Neural Radiance Fields (NeRF) to reconstruct the environment and generate synthetic training data.

Overview of the proposed pipeline: from aerial images to task learning.

1. Reconstruction with Road Priors

The process begins with a sequence of aerial images and their associated poses. A NeRF model is trained to represent the 3D scene. To mitigate issues like occlusions from trees or buildings—common in aerial views—the authors incorporate "road priors." These priors provide the model with predefined information regarding road geometry and color, ensuring that synthesized ground-level views remain accurate and free of artifacts.

2. Data Generation and Task Learning

Once the NeRF is optimized, it acts as a high-fidelity simulator. Thousands of synthetic ground-level images are rendered along various driving trajectories. This data is then used for two primary tasks:

  • Visuo-motor Policy Training: A convolutional neural network (CNN) is trained to map raw pixels to steering commands, using a pure-pursuit controller to provide ground-truth labels for the synthetic paths.
  • Visual Localization: The model learns to estimate the vehicle's coordinates and heading (yaw) from a single ground-level image.
The algorithmic procedure for environment reconstruction and NeRF training.

Experimental Setup

The framework was evaluated in a 1/10th scale "mini-city" environment. Aerial data consisted of approximately 100 images captured by a Parrot Bebop 2 drone. For evaluation, an Optitrack motion capture system provided ground-truth positioning. The deployment platform was a custom racecar equipped with an NVIDIA Jetson Xavier NX and a ZED 2i stereo camera.

Results

The aerial-trained models were compared against baselines trained on real human-driven ground data.

  • Driving Performance: In end-to-end control tests, the proposed method significantly reduced the need for human intervention. While a standard imitation learning baseline required 10 interventions to complete a lap, the aerial-trained model required only one.
  • Localization Accuracy: When fused with an Inertial Measurement Unit (IMU), the system achieved a position error of 0.09m to 0.12m and a heading error of 1.12° to 2.26°. Notably, the synthetic data often led to higher accuracy than real ground-view imagery, likely due to the reduced noise and increased consistency of NeRF-generated views.

Conclusion

This work demonstrates that NeRFs can serve as powerful, compact simulators for bridging the gap between aerial observation and ground-level action. By generating "off-nominal" data—such as views from incorrect lanes—the system teaches robots to recover from errors more effectively than traditional methods. Future research will focus on extending this framework to dynamic environments with moving actors and scaling to full-sized urban environments using satellite imagery.

Summary updated

Abstract

In this work, we consider the problem of learning end to end perception to control for ground vehicles solely from aerial imagery. Photogrammetric simulators allow the synthesis of novel views through the transformation of pre-generated assets into novel views.However, they have a large setup cost, require careful collection of data and often human effort to create usable simulators. We use a Neural Radiance Field (NeRF) as an intermediate representation to synthesize novel views from the point of view of a ground vehicle. These novel viewpoints can then be used for several downstream autonomous navigation applications. In this work, we demonstrate the utility of novel view synthesis though the application of training a policy for end to end learning from images and depth data. In a traditional real to sim to real framework, the collected data would be transformed into a visual simulator which could then be used to generate novel views. In contrast, using a NeRF allows a compact representation and the ability to optimize over the parameters of the visual simulator as more data is gathered in the environment. We demonstrate the efficacy of our method in a custom built mini-city environment through the deployment of imitation policies on robotic cars. We additionally consider the task of place localization and demonstrate that our method is able to relocalize the car in the real world.
BibTeX
@inproceedings{murali2024learning,
  author = {Varun Murali and Guy Rosman and Sertac Karaman and Daniela Rus},
  title = {{Learning autonomous driving from aerial imagery}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
}
Figure