Learning Controls Using Cross-Modal Representations: Bridging Simulation and Reality for Drone Racing

Rogerio Bonatti, Ratnesh Madaan, Vibhav Vineet, Sebastian A. Scherer, Ashish Kapoor

Abstract

Machines are a long way from robustly solving open-world perception-control tasks, such as first-person view (FPV) aerial navigation. While recent advances in end-to-end Machine Learning, especially Imitation and Reinforcement Learning appear promising, they are constrained by the need of large amounts of difficult-to-collect labeled real-world data. Simulated data, on the other hand, is easy to generate, but generally does not render safe behaviors in diverse real-life scenarios. In this work we propose a novel method for learning robust visuomotor policies for real-world deployment which can be trained purely with simulated data. We develop rich state representations that combine supervised and unsupervised environment data. Our approach takes a cross-modal perspective, where separate modalities correspond to the raw camera data and the system states relevant to the task, such as the relative pose of gates to the drone in the case of drone racing. We feed both data modalities into a novel factored architecture, which learns a joint low-dimensional embedding via Variational Auto Encoders. This compact representation is then fed into a control policy, which we trained using imitation learning with expert trajectories in a simulator. We analyze the rich latent spaces learned with our proposed representations, and show that the use of our cross-modal architecture significantly improves control policy performance as compared to end-to-end learning or purely unsupervised feature extractors. We also present real-world results for drone navigation through gates in different track configurations and environmental conditions. Our proposed method, which runs fully onboard, can successfully generalize the learned representations and policies across simulation and reality, significantly outperforming baseline approaches.

Introduction

Training autonomous drones for high-speed navigation, such as first-person view (FPV) racing, presents a significant challenge in robotics. While end-to-end learning from pixels to motor commands is promising, it suffers from the "sim-to-real gap"—where models trained in simulation fail in the complex, noisy real world. Conversely, classical modular pipelines are often brittle; if a detector fails or a physics model is inaccurate, the system crashes. This work introduces a method to bridge this gap by learning robust state representations that generalize across domains.

Overview of the Sim-to-Real direct transfer approach across various real-world environments.

Methodology: Cross-Modal Variational Autoencoders (CM-VAE)

The core contribution is the Cross-Modal Variational Autoencoder (CM-VAE). Instead of training a model to map pixels directly to commands, the researchers force the network to find a shared "latent language" between two distinct modalities:

  1. Raw FPV RGB images from the drone's camera.
  2. Geometric gate poses (relative distance, azimuth, polar angle, and yaw).
The CM-VAE architecture learns a shared latent space z by reconstructing both the input image and the gate pose.

By forcing two separate encoders to map these inputs into a shared low-dimensional latent space z, the network learns a representation that captures task-relevant geometry while ignoring visual "distractions" like lighting, grass color, or background textures. Once the CM-VAE is trained, the encoder is frozen, and a control policy is trained via Imitation Learning (Behavior Cloning) to map the latent vector z to velocity commands V using expert trajectories generated in simulation.

Experimental Setup and Datasets

The authors utilized the AirSim simulator to generate 300,000 image-pose pairs for CM-VAE training and 17,500 images with expert velocity commands for the policy. To further bridge the reality gap, 20,000 unlabeled real-world images were incorporated into the training. Because the CM-VAE can process unlabeled data in one modality, it can learn real-world visual styles without requiring manual gate labeling, effectively regularizing the latent space for real-world deployment.

Key Results

Simulation Robustness

In tracks with high randomized noise (e.g., 3m gate offsets), the CM-VAE approach maintained a 40% success rate, while standard end-to-end learning methods dropped to near zero. This suggests the cross-modal requirement prevents the model from overfitting to specific visual patterns.

Real-World Performance

The policy, trained purely on simulated data, successfully navigated real-world S-tracks and circular circuits. In circuit tests, the CM-VAE policy averaged 14.3 gates before failure, a massive improvement over direct pose regression, which averaged only 1.2 gates.

Environmental Resilience

The system demonstrated remarkable robustness to environmental conditions that typically baffle vision-based systems. It remained functional under wind gusts of up to 20 km/h and during heavy snowfall, proving that the geometric latent representation is significantly more stable than raw pixel-based approaches.

Conclusion and Future Work

By leveraging cross-modal representations, this work demonstrates that robust navigation policies can be learned in simulation and transferred directly to reality. The shared latent space acts as a powerful regularizer, ensuring the drone focuses on critical geometric features rather than domain-specific visual noise. Future research directions include incorporating temporal logic (e.g., RNNs) to handle momentary occlusions and optimizing for higher-speed racing maneuvers.

Summary updated

Abstract

Machines are a long way from robustly solving open-world perception-control tasks, such as first-person view (FPV) aerial navigation. While recent advances in end-to-end Machine Learning, especially Imitation and Reinforcement Learning appear promising, they are constrained by the need of large amounts of difficult-to-collect labeled real-world data. Simulated data, on the other hand, is easy to generate, but generally does not render safe behaviors in diverse real-life scenarios. In this work we propose a novel method for learning robust visuomotor policies for real-world deployment which can be trained purely with simulated data. We develop rich state representations that combine supervised and unsupervised environment data. Our approach takes a cross-modal perspective, where separate modalities correspond to the raw camera data and the system states relevant to the task, such as the relative pose of gates to the drone in the case of drone racing. We feed both data modalities into a novel factored architecture, which learns a joint low-dimensional embedding via Variational Auto Encoders. This compact representation is then fed into a control policy, which we trained using imitation learning with expert trajectories in a simulator. We analyze the rich latent spaces learned with our proposed representations, and show that the use of our cross-modal architecture significantly improves control policy performance as compared to end-to-end learning or purely unsupervised feature extractors. We also present real-world results for drone navigation through gates in different track configurations and environmental conditions. Our proposed method, which runs fully onboard, can successfully generalize the learned representations and policies across simulation and reality, significantly outperforming baseline approaches. Supplementary video: https://youtu.be/VKc3A5HlUU8
BibTeX
@misc{bonatti2019learning,
  author = {Rogerio Bonatti and Ratnesh Madaan and Vibhav Vineet and Sebastian A. Scherer and Ashish Kapoor},
  title = {{Learning Controls Using Cross-Modal Representations: Bridging Simulation and Reality for Drone Racing}},
  year = {2019},
  eprint = {1909.06993},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1909.06993},
}
Figure