TL;DR
FalconGym 2.0 introduces a photorealistic simulation framework using editable Gaussian Splatting (GSplat) to solve the "overfitting" problem in visual drone navigation. By using a novel Performance-Guided Refinement (PGR) algorithm, the system automatically generates challenging training tracks to create a single visual policy that generalizes to unseen environments and achieves a 98.6% success rate in zero-shot sim-to-real hardware tests.
What Problem Does This Paper Address?
Training drones to navigate complex racing tracks using only visual input (cameras) is notoriously difficult. While photorealistic simulators have enabled "sim-to-real" transfer, most current visual policies suffer from overfitting. They tend to memorize the specific background or the exact geometry of a single training track. If a gate is moved slightly or a new track layout is introduced, these policies often fail.
To combat this, researchers usually have to collect new data and retrain a policy for every single new track—a process that is time-consuming and computationally expensive. The authors of this paper sought to create a single, robust policy capable of navigating a "family" of tracks it has never seen before, including tracks with moving obstacles.
Methods
The authors present three core innovations:
- FalconGym 2.0 & Edit API: Moving away from slower NeRF-based rendering, the team built a framework on 3D Gaussian Splatting (GSplat). They developed an Edit API that allows for programmatic, world-frame modifications of the environment (adding, moving, rotating, or scaling gates) in milliseconds. This turns a static 3D scan into a dynamic, infinite training ground.
- Performance-Guided Refinement (PGR): Instead of training on random tracks, the PGR algorithm uses a min-max optimization approach. It identifies "weak spots" where the current policy performs poorly, uses the Edit API to synthesize similar but challenging tracks, and then refines the policy using imitation learning from a state-based expert.
- Modular Architecture: To prevent the model from memorizing the background, the authors decoupled perception from control.
- Perception: A U-Net predicts a binary mask of the gates from RGB images.
- Control: A lightweight controller consumes the mask and a short history of past actions to predict the next move. This modularity is key for deployment on low-power drone hardware.
Datasets
The researchers used a 3-minute video of a real-world flying arena to reconstruct the base environment. From this, they generated several synthetic benchmarks:
- Spatial-S: A track requiring sharp turns and altitude changes.
- Random: A track with alternating left/right gates.
- Moving: A dynamic track where gates translate in real-time.
- Real-World Hardware: Evaluation was performed on a ModalAI Starling 2 quadrotor (280g) equipped with a VOXL 2 onboard computer.
Key Insights and Ideas
- Editability as a Superpower: The ability to programmatically "nudge" the environment allows the trainer to act like an adversary, constantly finding the exact scenarios that cause the drone to fail and forcing it to learn.
- The Power of Two Gates: The authors found that training primarily on randomized two-gate segments was sufficient. Because the policy learns the local geometric relationship between gates, it can generalize to much longer, complex tracks through simple composition.
- GSplat vs. NeRF: By switching to GSplat, the researchers achieved significantly faster rendering and easier scene manipulation, which is critical for the iterative loops required by the PGR algorithm.
- Sim-to-Real via Masking: By training the controller on binary masks rather than raw RGB, the "reality gap" is narrowed. The controller doesn't care if the real-world wall is a different color than the sim wall; it only cares about the shape of the gate mask.
Key Results
- Generalization: While state-of-the-art baselines dropped to 0%–50% success on unseen tracks, the PGR-trained policy maintained a 100% success rate across all simulated test tracks without any per-track retraining.
- Robustness: In "gate-pose perturbation" tests (where gates are randomly shifted), the PGR policy tracked the performance of a perfect state-based expert much more closely than standard uniform sampling methods.
- Sim-to-Real Success: When deployed zero-shot on real hardware, the policy achieved a 98.6% success rate (69 out of 70 gates) across static and moving gate tracks.
- Efficiency: The Edit API operations finish in approximately 0.004 seconds on a modern GPU, enabling massive data augmentation on the fly.
Limitations and Future Work
- Observability: The current system assumes the next gate is always within the camera's field of view. If a track has a turn so sharp that the next gate is hidden, the policy may fail. The authors plan to relax this assumption in future versions.
- Dynamics Complexity: The study used standard Dubins airplane and quadrotor models. Incorporating more complex, high-speed aerodynamic effects could further improve high-speed racing performance.
- Onboard Integration: While the perception system runs onboard, the authors aim to further distill the entire two-network stack (perception + control) to run even more efficiently on resource-constrained flight controllers.
Summary updated