Reinforcement Learning based Autonomous Multi-Rotor Landing on Moving Platforms

Pascal Goldschmid, Aamir Ahmad

TL;DR

This paper presents a novel reinforcement learning (RL) approach for autonomous multi-rotor UAV landing on moving platforms. By decomposing the complex 3D landing into simpler 1D tasks and employing a curriculum learning strategy with transfer learning, the authors achieve significantly higher success rates and reduced training times compared to existing methods, even demonstrating real-world deployment.

What Problem Does This Paper Address?

Landing a multi-rotor Unmanned Aerial Vehicle (UAV) on a moving platform is crucial for extending their operational range and utility (e.g., for battery replenishment or data offloading). However, traditional control methods struggle with the inherent complexities and uncertainties of dynamic environments, often requiring highly accurate models that are difficult to derive. While Reinforcement Learning (RL) offers a data-driven alternative, existing RL approaches for this task suffer from long training times, inconsistent success rates, and a tedious trial-and-error process for hyperparameter tuning. This paper aims to overcome these limitations by developing a more efficient, robust, and interpretable RL-based landing system.

Methods

The authors introduce a multi-faceted approach to tackle the landing problem:

  • 1D Task Decomposition: The 3D landing problem is broken down into two independent 1D landing tasks (longitudinal and lateral) by using two instances of the same RL agent. Vertical and yaw movements are handled by traditional PID controllers. This simplification significantly reduces the complexity of the learning problem.
  • Curriculum Learning with Transfer Learning: The landing task is structured as a sequential curriculum. Each step in the curriculum involves learning a simpler version of the 1D landing task. Crucially, knowledge (Q-tables) is transferred from one step to the next, accelerating learning and improving stability.
  • Novel State Space Discretization: A sophisticated state space discretization technique is employed. This method leverages the kinematic model of the moving platform to understand the state space topology and uses a multiresolution approach to refine the state representation as the agent progresses through the curriculum. This helps mitigate the "curse of dimensionality."
  • Interpretable Hyperparameter Derivation: The authors derive equations to determine key hyperparameters (like agent frequency and maximum pitch angle) based on the platform's kinematics and the desired maneuverability of the UAV. This provides a systematic and understandable way to set these crucial parameters, avoiding manual tuning.
  • Tabular RL (Double Q-Learning): The core learning algorithm used is Double Q-Learning, a tabular RL method. This choice avoids the complexities of deep neural networks (DNNs) in deep RL, such as architecture design and extensive hyperparameter tuning, making it suitable for less powerful hardware and offering better interpretability.

Datasets

The paper does not explicitly mention pre-existing datasets for training or evaluation. Instead, the authors rely on:

  • Simulated Environment: A custom simulation environment built using Gazebo and the RotorS simulator. This environment allows for controlled training and evaluation of the RL agent under various platform trajectories.
  • Real-World Experiments: The authors conduct flight experiments using a quadcopter and a moving platform, providing a real-world validation of their approach.

The paper effectively creates its own "dataset" through extensive simulations and real-world trials, comparing against established baselines.

Key Insights and Ideas

  • Decomposition is Key: Breaking down a complex 3D control problem into simpler 1D sub-problems is a highly effective strategy for RL, significantly reducing learning complexity and time.
  • Curriculum + Transfer Learning = Supercharged Learning: The sequential curriculum, combined with transferring learned knowledge between steps, dramatically accelerates the learning process and improves the robustness of the final policy.
  • Kinematics for Interpretability: Using the platform's kinematic model to derive hyperparameters provides a principled and understandable way to ensure the UAV's maneuverability, moving beyond arbitrary trial-and-error.
  • Tabular RL's Niche: For this specific problem, tabular RL (Double Q-Learning) proves to be a viable and advantageous choice, offering faster training and better interpretability compared to deep RL methods, especially when combined with their novel curriculum and discretization.

Key Results

  • Significantly Reduced Training Time: The proposed method requires approximately 80% less training time compared to a deep RL baseline (Rodriguez-Ramos et al., 2019).
  • Improved Success Rates: The RL agent achieves higher success rates, with improvements of up to +26% in certain scenarios compared to the baseline.
  • Comparable Performance to PI Controllers: In two comparison scenarios, the RL approach achieves performance comparable to a cascaded PI controller, a well-established control method.
  • Real-World Deployment: The algorithm was successfully deployed and demonstrated on real hardware, showcasing its practical applicability.
  • Robustness to Noise: The method shows good performance even with noisy observations, though a slight drop is observed compared to noiseless simulations.

Limitations and Future Work

  • Limitations:
    • The current approach focuses on 1D control for longitudinal and lateral movements, with vertical and yaw controlled by PID.
  • The simulation environment does not explicitly model aerodynamic ground effects, which can contribute to the sim-to-real gap.
  • Real-world experiments revealed some sensitivity to differences in UAV mass and low-level controller tuning, as well as potential issues with state estimation and communication bandwidth.
  • The success rate in real-world experiments for moving platforms showed a noticeable drop compared to static platforms, attributed to factors like UAV dynamics, controller tuning, and ground effects.
  • Future Work:
    • Extend the RL approach to control vertical movement and yaw angle.
  • Investigate other tabular RL methods like n-step Q(σ) learning or eligible traces.
  • Address more complex landing scenarios, such as on inclined or flying platforms.
  • Improve the sim-to-real transfer by varying controller gains during training or incorporating more realistic physics in the simulation.

Summary updated

Abstract

Multi-rotor UAVs suffer from a restricted range and flight duration due to limited battery capacity. Autonomous landing on a 2D moving platform offers the possibility to replenish batteries and offload data, thus increasing the utility of the vehicle. Classical approaches rely on accurate, complex and difficult-to-derive models of the vehicle and the environment. Reinforcement learning (RL) provides an attractive alternative due to its ability to learn a suitable control policy exclusively from data during a training procedure. However, current methods require several hours to train, have limited success rates and depend on hyperparameters that need to be tuned by trial-and-error. We address all these issues in this work. First, we decompose the landing procedure into a sequence of simpler, but similar learning tasks. This is enabled by applying two instances of the same RL based controller trained for 1D motion for controlling the multi-rotor's movement in both the longitudinal and the lateral direction. Second, we introduce a powerful state space discretization technique that is based on i) kinematic modeling of the moving platform to derive information about the state space topology and ii) structuring the training as a sequential curriculum using transfer learning. Third, we leverage the kinematics model of the moving platform to also derive interpretable hyperparameters for the training process that ensure sufficient maneuverability of the multi-rotor vehicle. The training is performed using the tabular RL method Double Q-Learning. Through extensive simulations we show that the presented method significantly increases the rate of successful landings, while requiring less training time compared to other deep RL approaches. Finally, we deploy and demonstrate our algorithm on real hardware. For all evaluation scenarios we provide statistics on the agent's performance.
BibTeX
@misc{goldschmid2023reinforcement,
  author = {Pascal Goldschmid and Aamir Ahmad},
  title = {{Reinforcement Learning based Autonomous Multi-Rotor Landing on Moving Platforms}},
  year = {2023},
  eprint = {2302.13192},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.13192},
}
Figure