Abstract
Machines are a long way from robustly solving open-world perception-control tasks, such as first-person view (FPV) aerial navigation. While recent advances in end-to-end Machine Learning, especially Imitation and Reinforcement Learning appear promising, they are constrained by the need of large amounts of difficult-to-collect labeled real-world data. Simulated data, on the other hand, is easy to generate, but generally does not render safe behaviors in diverse real-life scenarios. In this work we propose a novel method for learning robust visuomotor policies for real-world deployment which can be trained purely with simulated data. We develop rich state representations that combine supervised and unsupervised environment data. Our approach takes a cross-modal perspective, where separate modalities correspond to the raw camera data and the system states relevant to the task, such as the relative pose of gates to the drone in the case of drone racing. We feed both data modalities into a novel factored architecture, which learns a joint low-dimensional embedding via Variational Auto Encoders. This compact representation is then fed into a control policy, which we trained using imitation learning with expert trajectories in a simulator. We analyze the rich latent spaces learned with our proposed representations, and show that the use of our cross-modal architecture significantly improves control policy performance as compared to end-to-end learning or purely unsupervised feature extractors. We also present real-world results for drone navigation through gates in different track configurations and environmental conditions. Our proposed method, which runs fully onboard, can successfully generalize the learned representations and policies across simulation and reality, significantly outperforming baseline approaches.
Introduction
Training autonomous drones for high-speed navigation, such as first-person view (FPV) racing, presents a significant challenge in robotics. While end-to-end learning from pixels to motor commands is promising, it suffers from the "sim-to-real gap"—where models trained in simulation fail in the complex, noisy real world. Conversely, classical modular pipelines are often brittle; if a detector fails or a physics model is inaccurate, the system crashes. This work introduces a method to bridge this gap by learning robust state representations that generalize across domains.
Methodology: Cross-Modal Variational Autoencoders (CM-VAE)
The core contribution is the Cross-Modal Variational Autoencoder (CM-VAE). Instead of training a model to map pixels directly to commands, the researchers force the network to find a shared "latent language" between two distinct modalities:
- Raw FPV RGB images from the drone's camera.
- Geometric gate poses (relative distance, azimuth, polar angle, and yaw).
By forcing two separate encoders to map these inputs into a shared low-dimensional latent space z, the network learns a representation that captures task-relevant geometry while ignoring visual "distractions" like lighting, grass color, or background textures. Once the CM-VAE is trained, the encoder is frozen, and a control policy is trained via Imitation Learning (Behavior Cloning) to map the latent vector z to velocity commands V using expert trajectories generated in simulation.
Experimental Setup and Datasets
The authors utilized the AirSim simulator to generate 300,000 image-pose pairs for CM-VAE training and 17,500 images with expert velocity commands for the policy. To further bridge the reality gap, 20,000 unlabeled real-world images were incorporated into the training. Because the CM-VAE can process unlabeled data in one modality, it can learn real-world visual styles without requiring manual gate labeling, effectively regularizing the latent space for real-world deployment.
Key Results
Simulation Robustness
In tracks with high randomized noise (e.g., 3m gate offsets), the CM-VAE approach maintained a 40% success rate, while standard end-to-end learning methods dropped to near zero. This suggests the cross-modal requirement prevents the model from overfitting to specific visual patterns.
Real-World Performance
The policy, trained purely on simulated data, successfully navigated real-world S-tracks and circular circuits. In circuit tests, the CM-VAE policy averaged 14.3 gates before failure, a massive improvement over direct pose regression, which averaged only 1.2 gates.
Environmental Resilience
The system demonstrated remarkable robustness to environmental conditions that typically baffle vision-based systems. It remained functional under wind gusts of up to 20 km/h and during heavy snowfall, proving that the geometric latent representation is significantly more stable than raw pixel-based approaches.
Conclusion and Future Work
By leveraging cross-modal representations, this work demonstrates that robust navigation policies can be learned in simulation and transferred directly to reality. The shared latent space acts as a powerful regularizer, ensuring the drone focuses on critical geometric features rather than domain-specific visual noise. Future research directions include incorporating temporal logic (e.g., RNNs) to handle momentary occlusions and optimizing for higher-speed racing maneuvers.
Summary updated