Reactive Aerobatic Flight via Reinforcement Learning

Zhichao Han, Xijie Huang, Zhoufeng Xu, J. Zhang, Yuze Wu, Mingyang Wang, Tianyue Wu, Fei Gao

Abstract

Traditional approaches to autonomous aerobatic flight rely on a decoupled architecture of trajectory optimization and path tracking, which often fails under the high-dynamic constraints and environmental uncertainties of extreme maneuvers. This paper presents an end-to-end reinforcement learning (RL) framework that maps sensor data directly to low-level motor commands. By utilizing a biased reset-based curriculum and a body-rate action space, the system achieves unprecedented reactivity. Experimental results demonstrate the first autonomous quadrotor capable of navigating moving gates while maintaining inverted flight, outperforming state-of-the-art optimization methods in both speed and tracking precision.

Introduction

Aerobatic flight represents the pinnacle of agile robotics, requiring a vehicle to operate at the very edge of its physical envelope. While human pilots can perform complex maneuvers like barrel rolls and power loops with fluid reactivity, autonomous systems have historically struggled with the latency and modeling errors inherent in multi-stage planning pipelines. This research introduces a model-free reinforcement learning approach that replaces traditional planning with a single, high-frequency neural network policy, enabling "reactive" aerobatics that can adapt to dynamic environments in real-time.

The Limitations of Traditional Planning

Conventional aerobatic control typically treats flight as a two-stage problem: first, optimizing a trajectory based on idealized physics, and second, employing a controller to track that path. This paradigm suffers from three critical vulnerabilities:

  1. Tracking Divergence: Real-world factors such as aerodynamic turbulence, battery voltage sag, and motor noise are difficult to model, causing the drone to drift from the pre-computed path during high-G maneuvers.
  2. Computational Latency: Re-calculating a trajectory in response to a moving target (e.g., a moving gate) can take seconds, which is orders of magnitude too slow for the millisecond-level adjustments required in aerobatics.
  3. Optimization Local Minima: Trajectory optimizers are sensitive to initial conditions and often converge on sub-optimal solutions that do not exploit the full agility of the airframe.

Methodology

Reinforcement Learning Framework

The core of the system is a policy trained via Proximal Policy Optimization (PPO). Unlike previous methods that output high-level position setpoints, this policy outputs collective thrust and body rates. This specific action space provides a "Goldilocks" balance: it is high-level enough to be robust against sim-to-real noise, yet low-level enough to allow the agent to exploit the quadrotor's maximum angular acceleration.

Automated Curriculum Learning

Learning aerobatics from scratch is a "sparse reward" problem; a drone is statistically unlikely to complete a barrel roll through random exploration. To solve this, the authors implemented Biased Reset-Based Training. Using the principle of differential flatness, the system analytically identifies states along a desired maneuver and starts the agent near the goal. As the agent's performance improves, the starting points are moved progressively further back, effectively "guiding" the policy through the complex physics of the maneuver.

Sim-to-Real Transfer

To bridge the gap between simulation (using the Flightmare engine) and reality, the researchers employed extensive domain randomization. They perturbed air drag coefficients, mass distributions, and control latencies during training. Furthermore, system delay was mitigated by averaging network outputs over a sliding window of past time steps, ensuring stable control on physical hardware.

Experimental Results

Performance Benchmarks

The RL-based policy was evaluated against a state-of-the-art optimization baseline across three benchmark tracks: the Power Loop, Barrel Roll, and Split-S. The RL agent completed these tracks 27% faster on average (12.3s vs 16.9s). More importantly, the RL approach maintained significantly lower position and attitude errors, demonstrating that an end-to-end policy can find more efficient and robust control laws than manual optimization.

Reactivity and Robustness

The system's reactivity was tested by moving waypoints during flight. The drone maintained a 78% success rate even when targets moved at 3.4 m/s, a speed significantly higher than any dynamic element encountered during the training phase. In real-world hardware tests using a Jetson Orin NX-equipped quadrotor, the system successfully navigated a moving gate while inverted—a first in the field of autonomous flight. The maximum tracking error recorded was 0.35m, compared to 0.95m for the baseline.

Discussion and Future Work

While this framework achieves a response time of just 0.3 ms, several challenges remain. The current implementation relies on an external Motion Capture System (MCS) for state estimation; transitioning to purely onboard vision-based sensing is a necessary next step for field deployment. Additionally, while the execution of maneuvers is autonomous, the "intention" (the sequence of waypoints) is still human-defined. Future research may explore the use of Generative AI to autonomously design and chain together novel aerobatic maneuvers, pushing the boundaries of robotic creativity and agility.

Summary updated

Abstract

Quadrotors have demonstrated the versatility, yet their full aerobatic potential remains largely untapped due to inherent underactuation and the complexity of aggressive maneuvers. Traditional approaches, separating trajectory optimization and tracking control, suffer from tracking inaccuracies, computational latency, and sensitivity to initial conditions, limiting their effectiveness in dynamic, high-agility scenarios. Inspired by recent advances in data-driven methods, we propose a reinforcement learning-based framework that directly maps drone states and aerobatic targets to control commands, eliminating modular separation to enable quadrotors to perform end-to-end policy optimization for extreme aerobatic maneuvers. To ensure efficient and stable training, we introduce an automated curriculum learning strategy that dynamically adjusts aerobatic task difficulty. Enabled by domain randomization for robust zero-shot sim-to-real transfer, our approach is validated in demanding real-world experiments, including the demonstration of an autonomous drone continuously performing inverted flight while reactively navigating a moving gate.
BibTeX
@article{han2025reactive,
  author = {Zhichao Han and Xijie Huang and Zhoufeng Xu and J. Zhang and Yuze Wu and Mingyang Wang and Tianyue Wu and Fei Gao},
  title = {{Reactive Aerobatic Flight via Reinforcement Learning}},
  journal = {IEEE Robotics and Automation Letters},
  year = {2025},
  doi = {10.1109/lra.2025.3606383},
}
Figure