Model-Based Meta-Reinforcement Learning for Flight with Suspended Payloads

Suneel Belkhale, Rachel Li, Gregory Kahn, Rowan McAllister, Roberto Calandra, Sergey LevineIEEE RA-L 2020 · DOI · OpenReview

Abstract

Flying quadcopters with suspended payloads presents a significant challenge due to the complex, hybrid dynamics introduced by the swinging mass. Traditional control methods often require precise knowledge of physical parameters, such as cable length and payload mass, which are difficult to estimate in real-time. This paper presents a model-based meta-reinforcement learning (MB-MRL) framework that enables quadcopters to adapt to unknown suspended payloads during flight. By augmenting a dynamics model with latent variables updated via variational inference, the system can identify the physics of a new payload within seconds. Experimental results demonstrate that this approach significantly outperforms non-adaptive controllers and generalizes effectively to unseen payload configurations.

Introduction

While quadcopter control is a well-studied field, the addition of a payload attached via a flexible cable transforms the vehicle into a complex system with high-order dynamics. The payload introduces oscillations and forces that vary drastically depending on the cable's length and the object's mass. Standard control strategies, such as PID or fixed-model System Identification, typically rely on a priori knowledge of these parameters. When a drone encounters a payload with an unknown weight or a different string length, the resulting model mismatch often leads to poor tracking performance or catastrophic instability.

The core challenge addressed in this work is rapid online adaptation. To operate effectively in dynamic environments, a robot must be able to identify how a new payload affects its flight characteristics and adjust its motor commands almost instantaneously without manual recalibration.

Methodology

The proposed framework utilizes a model-based meta-reinforcement learning approach, building upon the Probabilistic Ensembles with Trajectory Sampling (PETS) algorithm. The architecture consists of three primary components:

1. Latent Dynamics Model

The researchers trained a neural network to predict the next state ($s_{t+1}$) based on the current state and action. To account for varying physical properties, they introduced a stochastic latent variable ($z$) as an input. This $z$ variable serves as a "context" or "style" embedding that captures the specific dynamics of the current payload, such as its mass or the tether length, without requiring these values to be explicitly measured.

2. Meta-Training

During the training phase, the robot is exposed to a variety of payloads. The model learns a shared set of weights ($\theta$) that represent the general physics of flight, while simultaneously learning to map specific flight trajectories to the appropriate latent variable $z$. This "learning to learn" phase ensures the model is prepared for a wide distribution of possible dynamics.

3. Online Adaptation via Variational Inference

At test time, the robot maintains a small buffer of recent flight data. Every few milliseconds, the system performs variational inference to estimate the posterior distribution of $z$. By performing gradient descent on the Evidence Lower Bound (ELBO), the model asks: "Given the movements I just made and the resulting states, what kind of payload must I be carrying?" This updated $z$ is then fed into a Model Predictive Controller (MPC) to plan optimal future actions.

Experimental Evaluation

The system was evaluated using a DJI Tello quadcopter. The training dataset consisted of approximately 16,000 data points (roughly 1.1 hours of flight time) collected by a human pilot. The training data included payloads with tether lengths of 18cm and 30cm and masses ranging from 10 to 15 grams.

The evaluation tasks required the quadcopter to follow circle, square, and figure-8 trajectories in a vertical plane. To test the robustness of the meta-learning approach, the researchers also evaluated the model on unseen tether lengths (21cm and 27cm) to determine if the system could interpolate between the dynamics learned during training.

Results and Discussion

Tracking Performance

The MB-MRL approach achieved an average tracking error of approximately 24 pixels. In comparison, standard non-adaptive MBRL methods frequently failed to complete the tasks, often drifting out of the camera's field of view. The meta-learning framework also outperformed manually tuned PID controllers, which struggled to compensate for the payload's oscillations.

The Value of Latent Representations

A key insight from the study was that the model performed better when allowed to "discover" its own latent variables ($z$) rather than being provided with explicit physical parameters like tether length. This suggests that the latent space captures complex, unmodeled nuances—such as air resistance or magnetic gripper interactions—that simple physical constants fail to represent.

Rapid Adaptation and Generalization

Visualization of the latent variable $z$ during flight showed that it converges to a stable value almost immediately after the drone begins moving. As $z$ stabilizes, the tracking error drops significantly, demonstrating a direct link between the model's understanding of the payload and its flight accuracy. Furthermore, the system successfully handled sudden discontinuities, such as picking up or dropping a payload, by flushing the data buffer and re-adapting to the new state within seconds.

Conclusion

This work demonstrates that model-based meta-reinforcement learning is a powerful tool for the control of complex, variable robotic systems. By using latent variables and online variational inference, quadcopters can adapt to the unpredictable dynamics of suspended payloads in real-time.

Future work will focus on moving beyond external camera tracking to onboard sensing (such as IMUs or computer vision), optimizing the computational overhead of the MPC planner for onboard deployment, and testing the system's resilience to external environmental factors like wind.

Summary updated

Abstract

Transporting suspended payloads is challenging for autonomous aerial vehicles because the payload can cause significant and unpredictable changes to the robot's dynamics. These changes can lead to suboptimal flight performance or even catastrophic failure. Although adaptive control and learning-based methods can in principle adapt to changes in these hybrid robot-payload systems, rapid mid-flight adaptation to payloads that have a priori unknown physical properties remains an open problem. We propose a meta-learning approach that "learns how to learn" models of altered dynamics within seconds of post-connection flight data. Our experiments demonstrate that our online adaptation approach outperforms non-adaptive methods on a series of challenging suspended payload transportation tasks. Videos and other supplemental material are available on our website: https://sites.google.com/view/meta-rl-for-flight
BibTeX
@article{belkhale2020model,
  author = {Suneel Belkhale and Rachel Li and Gregory Kahn and Rowan McAllister and Roberto Calandra and Sergey Levine},
  title = {{Model-Based Meta-Reinforcement Learning for Flight with Suspended Payloads}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2020},
  doi = {10.1109/lra.2021.3057046},
}
Figure