Abstract
Agile drone flight in dynamic environments requires both high-level strategic reasoning and low-level physical precision. This paper introduces High-MPC, a framework that leverages policy search to optimize high-level decision variables for a Model Predictive Control (MPC) scheme. By treating the selection of parameters like gate-crossing times as a latent variable inference problem, the authors achieve robust, real-time performance in complex scenarios, such as threading moving gates. The approach demonstrates a 94-100% success rate in simulation and transfers seamlessly to real-world hardware.
Introduction
Navigating a quadrotor through a fast-moving gate presents a significant challenge: the optimal trajectory depends on the exact moment of crossing, yet the crossing time itself is a decision the controller must make. Standard Model Predictive Control (MPC) is highly effective at managing vehicle dynamics and hard constraints but often falters when faced with these high-level "scheduling" or combinatorial decisions. Conversely, Reinforcement Learning (RL) excels at learning complex strategies from data but can lack the safety and interpretability of model-based methods.
Researchers from the University of Zurich address this by proposing a hierarchical approach. Instead of replacing the controller with a neural network, they use a high-level policy to "guide" the MPC by providing it with optimized parameters, effectively bridging the gap between learning and control.
The High-MPC Framework
The core innovation of this work is the High-MPC architecture. In this setup, a high-level policy $\pi_\theta$ observes the environment state $o$ and outputs a set of decision variables $z$. These variables are then fed into an MPC optimizer.
As shown in the figure above, standard MPC (a) relies on fixed parameters, while other learning-based approaches (b) might try to learn the dynamics model itself. High-MPC (c) keeps the dynamics model known and precise, using the policy to adjust the optimization objective $\mathcal{L}(z)$ on-the-fly. For the gate-crossing task, $z$ typically includes the desired traversal time $t_i$ for each gate.
Policy Search via Probabilistic Inference
To train the high-level policy, the authors formulate the search for optimal parameters as a probabilistic inference problem. They utilize an Expectation-Maximization (EM) framework, which offers several advantages over traditional gradient-based RL:
- Stability: EM provides closed-form analytical updates for Gaussian policies, avoiding the sensitivity to learning rates often found in deep RL.
- Efficiency: The framework allows for three tiers of policy complexity: a constant Gaussian policy for fixed scenarios, a linear Gaussian policy for contextual adaptation, and a neural network policy for dynamic, real-time observations.
A key technical contribution is the "gate-following" cost function. This encourages the drone to align its velocity with the gate's motion as it approaches, providing a buffer against modeling errors in the gate's predicted trajectory.
Experimental Results
The framework was evaluated in both the Flightmare simulator and on physical hardware.
Simulation Performance
In scenarios involving three moving gates, High-MPC achieved a success rate of 94-100%. In contrast, a standard MPC baseline failed completely (0% success) because it could not resolve the timing conflicts inherent in moving targets. The system maintained high precision, with traversal errors limited to 17–30 cm even at high speeds.
Real-World Validation
The neural network policy, trained entirely in simulation on 40,000 samples, was deployed on a physical racing drone equipped with an Odroid XU4. Without any fine-tuning, the drone successfully navigated a moving pendulum gate. The computational efficiency is noteworthy: the policy inference takes less than 2ms, and the MPC optimization takes less than 5ms, enabling a 50Hz control loop suitable for agile maneuvers.
Conclusion
High-MPC demonstrates that the strengths of RL and MPC are complementary. By delegating high-level strategy to a learned policy and low-level execution to a model-based controller, the system achieves a level of agility and robustness previously difficult to attain. Future work aims to reduce the data intensity of training and integrate raw sensor inputs, such as vision or event-camera data, directly into the high-level policy to further reduce dependence on external state estimation systems.
Summary updated