Autonomous Blimp Control via H-infinity Robust Deep Residual Reinforcement Learning

Yang Zuo, Yu Tang Liu, Aamir AhmadarXiv 2023 · OpenReview

TL;DR

This paper introduces a novel H-infinity robust deep reinforcement learning (DRRL) controller for autonomous blimps, significantly improving stability and performance over prior PID-based methods. By integrating a robust H-infinity controller as a safety net, the system can handle complex dynamics and disturbances more effectively, enabling more aggressive exploration and better task completion.

What Problem Does This Paper Address?

Autonomous blimps offer superior energy efficiency and payload capacity for long-duration aerial tasks compared to quadcopters. However, their control is challenging due to complex dynamics, inherent modeling uncertainties, and susceptibility to environmental disturbances like wind. Existing solutions often struggle to balance robust stability with optimal performance. A previous approach combined deep reinforcement learning (DRL) with a Proportional-Integral-Derivative (PID) controller, but this PID controller's limited robustness could hinder the DRL agent's learning and performance, especially in complex scenarios. This paper aims to overcome these limitations by replacing the PID controller with a more robust H-infinity controller.

Methods

The core innovation is the integration of an H-infinity (H∞) robust controller with a deep residual reinforcement learning (DRRL) framework. Instead of a PID controller, an H∞ controller is used as the "base controller." This H∞ controller is designed to guarantee stability even under worst-case scenarios of model uncertainty and external disturbances.

A key element is the "mixer" block, which dynamically blends the actions of the H∞ base controller and the RL agent. Unlike previous work that used a fixed mixing factor, this paper introduces a variable mixing factor (denoted by 'q'). This factor allows the system to dynamically adjust the authority given to the RL agent. When disturbances are high, 'q' can be increased to give more control to the robust H∞ controller, ensuring safety. Conversely, when conditions are stable, 'q' can be decreased, allowing the RL agent more freedom to explore and improve performance.

The paper also highlights improvements in the blimp's control capabilities:

  • Thrust Vectoring: The base controller's thrust vectoring mechanism is enhanced to improve altitude control and enable more efficient descent.
  • Reverse Thrust: The use of reverse thrust is explicitly incorporated and managed to prevent misuse.

The DRRL agent is trained using Proximal Policy Optimization (PPO). The problem is formulated as a Markov Decision Process (MDP), with state and action spaces carefully defined for both a simplified TurtleSim environment and the more complex blimp simulator.

Datasets

The paper uses two simulation environments for training and evaluation:

  • TurtleSim: A simplified, toy environment used for initial testing and ablation studies. It's described as having no inherent dynamics, with disturbances simulated as time-dependent noise.
  • Blimp Simulator: A high-fidelity simulator implemented in ROS/Gazebo, which includes realistic fluid dynamics and supports software-in-the-loop (SITL) simulation. This simulator is used for more challenging tasks, including a "coil trajectory" with varying wind and buoyancy disturbances.

No specific external datasets are mentioned; the data is generated through interaction with these simulators.

Key Insights and Ideas

  • H∞ Robustness is Crucial for RL Stability: Replacing a PID with an H∞ controller provides a much stronger safety guarantee, allowing the RL agent to explore more aggressively without destabilizing the system. This is particularly important for blimps where instability can have severe consequences.
  • Variable Mixing Factor ('q') is Key to Balancing Performance and Safety: Dynamically adjusting the control authority between the robust base controller and the learning agent is critical. This allows the system to be safe under uncertainty and performant when conditions allow.
  • Thrust Vectoring and Reverse Thrust Unlock Performance: Efficiently utilizing thrust vectoring and managing reverse thrust are essential for achieving good altitude control and navigating challenging trajectories, especially for blimps which have slow inherent dynamics.
  • Training without Disturbance Can Be Beneficial: For the H∞-PPO agent, training without simulated wind disturbance actually led to better performance in the presence of wind during testing. This suggests that the robust base controller can handle disturbances, and training without them allows the RL agent to learn more aggressive, efficient behaviors.

Key Results

  • Improved Performance and Robustness: The H∞-PPO controller consistently outperformed the PID-PPO baseline in both TurtleSim and the blimp simulator. This was measured by a custom metric called "bscore," which considers time and energy efficiency.
  • Higher Success Rate: In the blimp simulator, the H∞-PPO controller achieved a significantly higher success rate, especially under challenging wind and buoyancy conditions, with fewer "fail" trials.
  • Effective Handling of Complex Trajectories: The H∞-PPO agent successfully navigated a challenging "coil trajectory" with varying wind disturbances, demonstrating its ability to handle complex maneuvers and environmental changes. For instance, in Table VI, the H∞-PPO had 0 failures across various wind and buoyancy conditions when q=0.5, while the PID-PPO had failures.
  • Variable 'q' Optimization: The paper found that a mixing factor of q=0.5 generally provided a good balance for testing, while training with a higher average 'q' improved stability.

Limitations and Future Work

  • Simulation-to-Real Gap: While the paper uses a high-fidelity simulator, the transition to real-world blimp control might still present challenges.
  • Computational Cost: H∞ controller design and DRRL training can be computationally intensive.
  • Future Work: The authors suggest further exploration of different mixing factor distributions for training and investigating more advanced control allocation strategies. They also imply that the robust framework could be extended to other underactuated aerial vehicles.

Summary updated

Abstract

Due to their superior energy efficiency, blimps may replace quadcopters for long-duration aerial tasks. However, designing a controller for blimps to handle complex dynamics, modeling errors, and disturbances remains an unsolved challenge. One recent work combines reinforcement learning (RL) and a PID controller to address this challenge and demonstrates its effectiveness in real-world experiments. In the current work, we build on that using an H-infinity robust controller to expand the stability margin and improve the RL agent's performance. Empirical analysis of different mixing methods reveals that the resulting H-infinity-RL controller outperforms the prior PID-RL combination and can handle more complex tasks involving intensive thrust vectoring. We provide our code as open-source at https://github.com/robot-perception-group/robust_deep_residual_blimp.
BibTeX
@misc{zuo2023autonomous,
  author = {Yang Zuo and Yu Tang Liu and Aamir Ahmad},
  title = {{Autonomous Blimp Control via H-infinity Robust Deep Residual Reinforcement Learning}},
  year = {2023},
  eprint = {2303.13929},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.13929},
}
Figure