Automated Aircraft Recovery via Reinforcement Learning: Initial Experiments

Jeffrey F. Monaco, David G. Ward, Andrew G. BartoNeurIPS 1997

Introduction

In high-agility flight, aircraft can occasionally enter "out-of-control" states where a pilot may become disoriented or lack the reaction time to recover safely. While standard flight controllers—such as Linear-Quadratic (LQ) trackers—are excellent at maintaining stability during normal flight, they often lack the "global" strategic awareness needed to navigate an aircraft from a steep dive or climb back to level flight without hitting the ground or exceeding structural and physiological limits.

To address this, researchers have proposed an Automated Recovery System (ARS) powered by Reinforcement Learning (RL). The challenge for this system is twofold: it must optimize the flight path to find the fastest return to straight-and-level flight, while strictly satisfying constraints to ensure the recovery maneuver does not subject the pilot to lethal G-forces or destroy the airframe.

Hierarchical Control Architecture

The researchers utilized a hierarchical control architecture that combines high-level AI decision-making with low-level traditional control:

  • Outer-Loop (The "Brain"): An RL agent selects longitudinal-stick commands (e.g., how hard to pull or push the stick).
  • Inner-Loop (The "Muscles"): A standard LQ tracking control algorithm translates the RL agent's desired pitch rate into actual actuator movements, such as adjusting elevons, flaps, and the rudder.

The system was trained using Residual Advantage Learning, an extension of Q-Learning that approximates an "advantage function" representing the value of taking a specific action relative to the optimal action. This approach proved more stable and performant than standard Q-Learning. To further improve training stability and reduce sensitivity to hyperparameters, the authors applied the Buckingham $\pi$-theorem from fluid dynamics to create dimensionless, normalized input features—effectively baking physical consistency into the learning process.

Training was conducted in a medium-fidelity, nonlinear, six-degree-of-freedom (6-DOF) F-16 simulation environment that included inertial cross-coupling and gravitational effects.

Key Insights from Training

During the development of the ARS, several critical insights emerged:

  • The "Pancaking" Problem: Standard controllers often fail because they lack ground awareness. If an aircraft is parallel to the earth but falling vertically (pancaking), a standard stabilizer might do nothing. The RL agent, however, successfully learned to pull up aggressively to arrest the descent.
  • Action Space Authority: Initial experiments with small stick inputs failed because the aircraft lacked the "authority" to pull out of steep dives. Expanding the discrete action set to include higher-magnitude stick forces was essential for successful recovery.
  • Constraint Integration: By adding specific features and continuing training, the RL agent learned to respect "soft" constraints, such as pilot G-limits, which are traditionally difficult to program into nonlinear controllers.

Recovery Performance and Results

The RL-based ARS demonstrated impressive performance in simulated out-of-control scenarios. In unconstrained tests, the agent successfully recovered the aircraft from extreme pitch offsets (up to $\pm 45^\circ$ from trim) to level flight in approximately 11 to 12 seconds.

Flight telemetry during a simulated recovery maneuver from a steep dive.

For the worst-case dive scenarios, the system minimized altitude loss to about 1,667 feet, which aligns with the physical limits of the aircraft's control authority. Furthermore, in constrained experiments, the RL agent successfully learned to keep the pilot-station normal acceleration within a safe range of -1.5g to +3.5g, a significant improvement over unconstrained agents that frequently exceeded these physiological limits.

Recovery time across various initial pitch attitudes, demonstrating consistent recovery within 12 seconds.

Limitations and Future Work

While these initial experiments are promising, the study focused almost exclusively on the pitch axis. Real-world out-of-control states often involve complex, multi-axis rolls and spins. Additionally, the agent was limited to choosing from a set of 9 discrete stick positions; future iterations would benefit from continuous action spaces for smoother control. Finally, the 72-hour training time for a single-axis recovery highlights the need for more efficient exploration strategies as the system scales to full 6-DOF, multi-axis recovery involving coordinated longitudinal and lateral stick sequences.

Summary updated

BibTeX
@inproceedings{monaco1997automated,
  author = {Jeffrey F. Monaco and David G. Ward and Andrew G. Barto},
  title = {{Automated Aircraft Recovery via Reinforcement Learning: Initial Experiments}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1997},
}
Figure