Deep Residual Reinforcement Learning based Autonomous Blimp Control

Yu Tang Liu, Eric Price, Michael J. Black, Aamir Ahmad

TL;DR

This paper introduces a novel Deep Residual Reinforcement Learning (DRRL) framework that combines a traditional PID controller with a reinforcement learning agent to achieve robust autonomous control for blimps. The DRRL agent learns to fine-tune PID control decisions, significantly improving performance and robustness to environmental disturbances like wind and changing buoyancy, even when trained solely in simulation.

What Problem Does This Paper Address?

Autonomous blimps offer unique advantages for long-duration aerial tasks due to their energy efficiency and quiet operation. However, controlling them is challenging. Blimps have complex, non-linear, and time-delayed dynamics, coupled with constantly changing buoyancy due to environmental factors. Traditional PID controllers struggle to handle these complexities, leading to large tracking errors. While reinforcement learning (RL) offers a promising alternative, training RL agents can be data-intensive and prone to learning unsafe behaviors. This paper aims to bridge this gap by developing a more stable and robust control system for blimps that leverages the strengths of both classical control and deep reinforcement learning.

Methods

The core of the paper is the Deep Residual Reinforcement Learning (DRRL) framework. This approach combines a pre-existing PID controller (acting as a "stability provider") with a Deep Reinforcement Learning agent (the "performance optimizer").

Here's a breakdown of the key methodological components:

  • DRRL Framework: The DRRL agent learns to modify the decisions of the PID controller. This is achieved by having the RL agent's output be a residual adjustment to the PID's output. This "residual" approach constrains the RL agent's search space, implicitly guiding it towards safer and more stable behaviors.
  • PID Controller: A standard PID controller is used to provide baseline performance and stability. It's designed to handle basic control tasks, but its limitations in dynamic environments are acknowledged.
  • LSTM Layer: To address the partial observability of the environment (e.g., wind, buoyancy), an LSTM (Long Short-Term Memory) layer is integrated into the policy network. This allows the agent to better remember past states and infer current environmental conditions.
  • Domain Randomization: To bridge the "sim-to-real" gap, the simulation environment is randomized with respect to various parameters (wind, buoyancy, etc.) during training. This helps the trained agent generalize better to real-world conditions.
  • Action Space Design: To prevent actuator chattering and ensure smoother control, the action space is designed to output increments to actuator commands rather than absolute values.
  • Hybrid Mixer: A novel "hybrid mixer" is proposed to combine the outputs of the PID controller and the RL agent. This mixer balances the authority of the RL agent with the stability of the PID, offering a more nuanced control strategy than purely absolute or relative mixing.
  • Software-in-the-Loop (SITL) Simulation: Training is performed in a simulated environment to accelerate the process and allow for extensive experimentation.

Datasets

The paper does not explicitly mention or introduce new datasets. Instead, it relies on:

  • Simulated Environment: A ROS/Gazebo SITL simulation environment is used for training and evaluating the DRRL agent. This simulation is designed to mimic the dynamics of a real blimp and includes domain randomization for robustness.
  • Real-World Experiments: The trained agent is evaluated on a physical blimp in outdoor, windy conditions.

Key Insights and Ideas

  • Leveraging Classical Control for RL Stability: The core insight is that by using a PID controller as a baseline and having the RL agent learn residuals to its output, the training process becomes significantly more stable and the agent is implicitly guided away from unsafe behaviors. This is a powerful way to combine the benefits of model-free RL with the guarantees of classical control.
  • LSTM for Environmental Robustness: The inclusion of an LSTM layer is crucial for handling the partially observable nature of blimp control, particularly the impact of wind and buoyancy. It allows the agent to build an internal model of the environment's state over time.
  • Sim-to-Real Generalization through Domain Randomization: The paper demonstrates that extensive domain randomization in simulation is effective in creating an agent that can generalize to real-world conditions, even with parameters not explicitly randomized during training.
  • Hybrid Mixer for Balanced Control: The hybrid mixer proves to be a more effective approach than purely absolute or relative mixing for combining PID and RL outputs, offering a good balance between performance improvement and training stability.

Key Results

  • Improved PID Performance: The DRRL agent consistently improves upon the baseline PID controller's performance in simulations, achieving better tracking and stability.
  • Yaw Control Ablation Study: In a simplified yaw control task, the DRRL agent with an LSTM layer showed a significant performance increase (47% for good PID, 13% for poor PID) compared to PID alone, and reduced performance drops during exploration.
  • Robustness to Disturbances: Rigorous simulations showed the DRRL agent to be robust to changes in wind speed and buoyancy.
  • Real-World Success: The agent, trained solely in simulation, successfully controlled a real blimp in windy conditions, demonstrating strong generalization capabilities.
  • Trajectory Tracking: The DRRL agent successfully tracked both square and coil trajectories in simulation, showing less overshoot than PID and baseline methods, especially in challenging coil trajectories.
  • Training Stability: The hybrid mixer provided better training stability compared to an absolute mixer, which sometimes led to the agent getting stuck in local optima.

Limitations and Future Work

  • Simulation-to-Real Gap: While domain randomization helps, the paper acknowledges that some real-world effects (e.g., trim weight differences, maximum thrust variations) were not explicitly randomized during training, posing a challenge for generalization.
  • Tuning Complexity: While the DRRL framework aims to reduce tuning complexity compared to pure RL, some parameters (e.g., reward weights, mixer parameters) still require careful tuning.
  • Future Work: The authors suggest exploring more advanced RL algorithms and further investigating the impact of different mixer designs. They also highlight the potential for extending this framework to other complex robotic systems.

Summary updated

Abstract

Blimps are well suited to perform long-duration aerial tasks as they are energy efficient, relatively silent and safe. To address the blimp navigation and control task, in previous work we developed a hardware and software-in-the-loop framework and a PID-based controller for large blimps in the presence of wind disturbance. However, blimps have a deformable structure and their dynamics are inherently non-linear and time-delayed, making PID controllers difficult to tune. Thus, often resulting in large tracking errors. Moreover, the buoyancy of a blimp is constantly changing due to variations in ambient temperature and pressure. To address these issues, in this paper we present a learning-based framework based on deep residual reinforcement learning (DRRL), for the blimp control task. Within this framework, we first employ a PID controller to provide baseline performance. Subsequently, the DRRL agent learns to modify the PID decisions by interaction with the environment. We demonstrate in simulation that DRRL agent consistently improves the PID performance. Through rigorous simulation experiments, we show that the agent is robust to changes in wind speed and buoyancy. In real-world experiments, we demonstrate that the agent, trained only in simulation, is sufficiently robust to control an actual blimp in windy conditions. We openly provide the source code of our approach at https://github.com/robot-perception-group/AutonomousBlimpDRL. Video demonstration is provided at https://youtu.be/EMC4KnlH0yI.
BibTeX
@inproceedings{liu2022deep,
  author = {Yu Tang Liu and Eric Price and Michael J. Black and Aamir Ahmad},
  title = {{Deep Residual Reinforcement Learning based Autonomous Blimp Control}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=pXstiuaS0V},
}
Figure