Control of a Quadrotor With Reinforcement Learning

Jemin Hwangbo, Inkyu Sa, Roland Siegwart, Marco Hutter

Introduction

Traditional quadrotor control typically relies on a "control stack" where high-level trajectory planners feed into low-level PID or Model Predictive Control (MPC) loops. While effective, these systems require significant manual tuning and predefined structures that can limit performance in extreme conditions. Researchers from ETH Zurich have developed a reinforcement learning (RL) approach that allows a quadrotor to be controlled entirely by a neural network, mapping raw sensor data directly to motor thrusts. The resulting policy is computationally efficient—evaluating in just 7 microseconds—and remarkably robust, capable of stabilizing a drone even after being thrown into the air upside-down at high speeds.

Methods

The authors introduce a deterministic on-policy learning algorithm designed for stability and efficiency. Key technical components include:

  • State Representation: Instead of using quaternions, which have a mathematical "pitfall" where two different values represent the same rotation, the model uses nine elements of a rotation matrix. This ensures a continuous and unique mapping for the neural network to learn.
  • Network Architecture: The system utilizes a relatively simple multilayer perceptron (MLP) with two hidden layers of 64 tanh nodes. One network estimates the "Value" (expected reward), and another determines the "Policy" (the action to take).
  • Natural Gradient Descent: To update the policy, a natural gradient approach with a Mahalanobis metric is used. This helps the algorithm make consistent progress without taking steps that are too large and destabilizing the controller.
  • Hybrid Training: During early training, a simple PD (Proportional-Derivative) controller is added to the RL output to prevent the simulation from crashing immediately. As the RL policy improves, it eventually learns behaviors far more sophisticated than the PD baseline.

Experimental Setup

Training was performed in the Robotic Artificial Intelligence (RAI) framework, a C++ environment built for speed. Real-world evaluation was conducted on an Ascending Technologies Hummingbird quadrotor equipped with an Intel Computer Stick for onboard computation and a Vicon motion capture system for state estimation.

Key Results and Insights

The RL policy demonstrated significant advantages over traditional methods and standard RL baselines:

  • Extreme Recovery: The drone successfully recovered from "harsh initialization"—being thrown by hand upside-down into the air at velocities of 5 m/s.
Figure 1: Time-lapse of the quadrotor recovering from an inverted throw using the learned RL policy.
  • Training Efficiency: The policy reached a high level of performance after only about 10 minutes of training and fully converged within 25 minutes. This is significantly faster than standard algorithms like DDPG or TRPO-gae.
Figure 5: Comparison of training performance (average sum of cost over time) between the proposed method and baselines.
  • Precision and Robustness: In waypoint tracking tests, the drone maintained a steady-state error of only 1.3 cm. In simulation tests with random initial states, the RL policy had a failure rate of only 4%, compared to 71% for a baseline linear MPC controller.
  • Computational Efficiency: The neural network policy takes only 7 µs to evaluate, compared to approximately 1,000 µs for a standard linear MPC. This massive speedup frees up onboard CPU resources for other tasks like computer vision or mapping.

Discussion and Future Work

A key insight from this research is that simplicity in modeling can be beneficial. The authors used a simulation model that ignored air drag and complex aerodynamics, finding that the resulting policy was more stable in the real world, as real-world drag naturally helps dampen motion.

However, some limitations remain. The steady-state error of 1.3 cm, while small, is higher than some specialized high-gain classical controllers. Additionally, the current model-free policy does not adapt to changing dynamics, such as battery voltage drops. Future work will explore Recurrent Neural Networks (RNNs) to help the controller adapt to modeling errors in real-time and investigate transfer learning to further bridge the "reality gap."

Summary updated

BibTeX
@article{hwangbo2017control,
  author = {Jemin Hwangbo and Inkyu Sa and Roland Siegwart and Marco Hutter},
  title = {{Control of a Quadrotor With Reinforcement Learning}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2017},
  doi = {10.1109/lra.2017.2720851},
}
Figure