Reinforcement-Learning-Based Disturbance Rejection Control for Uncertain Nonlinear Systems

Maopeng Ran, Juncheng Li, Lihua Xie

TL;DR

This paper introduces a novel approach to control uncertain nonlinear systems by combining Reinforcement Learning (RL) with Active Disturbance Rejection Control (ADRC). The method uses an Extended State Observer (ESO) to estimate and cancel out unknown disturbances in real-time, while an RL agent learns the optimal control policy for the now-compensated system, crucially without requiring restrictive excitation conditions.

What Problem Does This Paper Address?

Controlling complex, real-world systems often involves dealing with nonlinear dynamics and unpredictable external disturbances. Traditional control methods can struggle with these uncertainties, especially when the system's exact mathematical model is unknown or "non-simple." Reinforcement Learning (RL) offers a powerful way to learn optimal control policies from experience, but its application to nonlinear systems often requires specific conditions, like "persistence of excitation" (PE), which can degrade performance. This paper tackles the challenge of robustly controlling uncertain nonlinear systems by integrating the disturbance-handling capabilities of ADRC with the learning power of RL, while circumventing the limitations of traditional RL approaches.

Methods

The core of the proposed method lies in a two-pronged strategy:

  1. Extended State Observer (ESO): An ESO is employed to estimate not only the system's internal states but also the "total uncertainty." This total uncertainty encompasses all unknown dynamics and external disturbances acting on the system.
  2. Disturbance Compensation and RL-based Optimal Control:
    • The estimated total uncertainty from the ESO is used to actively cancel out disturbances in real-time. This effectively transforms the uncertain nonlinear system into a more predictable, "compensated" system.
  3. Simultaneously, a simulation of experience-based RL technique is used to learn the optimal control policy for this compensated system. This involves using an actor-critic neural network architecture to approximate the optimal value function and control policy.
  4. A key innovation is the use of "concurrent learning" (also known as experience replay) to extrapolate the Bellman error (a measure of how well the RL agent is learning) to a predefined set of data points. This allows the RL agent to learn effectively without the system needing to physically visit those points, thus relaxing the strict persistence of excitation (PE) condition.

Datasets

Not discussed in the paper. The paper relies on simulated data generated from the described nonlinear system models for evaluation.

Key Insights and Ideas

  • Synergy of ADRC and RL: The paper demonstrates a powerful synergy between ADRC's disturbance rejection capabilities and RL's ability to learn optimal policies, creating a robust control framework for complex systems.
  • Relaxing PE Condition: By using concurrent learning and extrapolating the Bellman error, the authors successfully remove the need for the restrictive persistence of excitation (PE) condition, which is a significant practical advantage for RL-based control.
  • Practicality of Output Feedback: The proposed method is output feedback-based, meaning it only requires the system's output for control, making it more applicable to real-world scenarios where full state information might not be available.
  • Handling Non-Simple Models: The framework is designed to handle "non-simple" nominal models, which are common in practice and can be challenging for traditional control design.

Key Results

The paper presents rigorous theoretical analysis demonstrating the practical convergence of the system state to the origin and the learned control policy to the ideal optimal policy. Simulation results are provided for two examples of uncertain nonlinear systems, showcasing:

  • Effective regulation of system states to the origin.
  • Accurate estimation of system states and total uncertainty by the ESO.
  • Convergence of the actor NN weights to their ideal values.
  • Good performance even with different data grid sizes for extrapolation, though more data generally leads to better convergence.
  • Successful application to both known and unknown basis functions for the RL agent.

Limitations and Future Work

  • Parameter Tuning: The developed framework involves several parameters for both the ESO and the RL components (e.g., adaptation gains, observer gains, saturation bounds). While the paper provides some guidance and sufficient conditions, selecting optimal parameters might still require some trial and error.
  • Practical Convergence: The theoretical results are for "practical convergence," meaning the cost evaluated along the approximate optimal trajectory is infinite. While this is a common theoretical construct in RL, it's a limitation in terms of achieving a truly finite optimal cost in practice.
  • Future Work: The authors suggest extending the results to systems with constraints and applying the developed approach to robotic systems.

Summary updated

Abstract

This article investigates the reinforcement-learning (RL)-based disturbance rejection control for uncertain nonlinear systems having nonsimple nominal models. An extended state observer (ESO) is first designed to estimate the system state and the total uncertainty, which represents the perturbation to the nominal system dynamics. Based on the output of the observer, the control compensates for the total uncertainty in real time, and simultaneously, online approximates the optimal policy for the compensated system using a simulation of experience-based RL technique. Rigorous theoretical analysis is given to show the practical convergence of the system state to the origin and the developed policy to the ideal optimal policy. It is worth mentioning that the widely used restrictive persistence of excitation (PE) condition is not required in the established framework. Simulation results are presented to illustrate the effectiveness of the proposed method.
BibTeX
@article{ran2021reinforcement,
  author = {Maopeng Ran and Juncheng Li and Lihua Xie},
  title = {{Reinforcement-Learning-Based Disturbance Rejection Control for Uncertain Nonlinear Systems}},
  journal = {IEEE Transactions on Cybernetics},
  year = {2021},
  doi = {10.1109/tcyb.2021.3060736},
}

Loading related content...

Figure