Abstract
This paper investigates the reinforcement learning (RL) based disturbance rejection control for uncertain nonlinear systems having non-simple nominal models. An extended state observer (ESO) is first designed to estimate the system state and the total uncertainty, which represents the perturbation to the nominal system dynamics. Based on the output of the observer, the control compensates for the total uncertainty in real time, and simultaneously, online approximates the optimal policy for the compensated system using a simulation of experience based RL technique. Rigorous theoretical analysis is given to show the practical convergence of the system state to the origin and the developed policy to the ideal optimal policy. It is worth mentioning that, the widely-used restrictive persistence of excitation (PE) condition is not required in the established framework. Simulation results are presented to illustrate the effectiveness of the proposed method.
Introduction
In modern control theory, achieving a balance between optimality and robustness remains a significant challenge. Reinforcement Learning (RL) has emerged as a powerful tool for finding optimal control policies by minimizing predefined cost functions. However, standard RL implementations often rely on the Persistence of Excitation (PE) condition—a requirement for continuous system perturbation to ensure sufficient data for learning. In many real-world engineering applications, such "wiggling" is impractical or even hazardous.
Conversely, Active Disturbance Rejection Control (ADRC) is renowned for its robustness. By utilizing an Extended State Observer (ESO), ADRC can estimate and cancel real-time disturbances and modeling errors. Yet, traditional ADRC often assumes a simplified "chain of integrators" model, potentially ignoring valuable known nominal physics. This paper proposes a hybrid framework that integrates RL with ADRC to achieve optimal control for complex nonlinear systems without requiring PE.
Methodology
The proposed architecture consists of two primary layers: an Observer Layer and a Control Layer.
The Observer Layer (ESO)
The Extended State Observer (ESO) is the cornerstone of the robustness strategy. It is designed to estimate not only the internal states of the system but also a "total uncertainty" term. This term encapsulates external disturbances, modeling mismatches, and unmodeled dynamics. By estimating these factors in real-time, the system can actively compensate for them.
The Control Layer
The control signal is bifurcated into two components. The first component utilizes the ESO's estimate to cancel out the total uncertainty, effectively "cleaning" the system dynamics. The second component is an Actor-Critic RL controller that focuses on optimizing the "nominal" (known) part of the system.
Simulation of Experience
To bypass the restrictive PE condition, the authors employ a "simulation of experience" (or concurrent learning) technique. Leveraging the known nominal model, the RL agent does not rely solely on the current physical trajectory. Instead, it extrapolates the Bellman error across a fixed grid of points in the state space. This allows the agent to learn from states it has not physically visited, ensuring the convergence of neural network weights without the need for injected noise.
Key Results
The effectiveness of the proposed method was validated through numerical simulations of complex nonlinear systems.
Convergence without PE: The simulations demonstrated that the actor neural network weights converged to their ideal values even in the absence of probing noise. This confirms that the "simulation of experience" successfully replaces the need for physical exploration.
Robustness and Accuracy: The controller successfully regulated systems to the origin despite significant modeling mismatches and time-varying disturbances. As shown in the state trajectories, the ESO provides highly accurate estimates ($\hat{x}$) of the true states ($x$), allowing the controller to maintain stability and performance.
Data Efficiency: Results indicated that a relatively sparse grid (e.g., $5 \times 5$) for simulating experience was sufficient for convergence, highlighting the efficiency of the extrapolation method.
Conclusion
This research presents a robust and optimal control framework for uncertain nonlinear systems. By combining the disturbance-rejection capabilities of ADRC with the optimality of RL, and utilizing a simulation-of-experience approach to eliminate the PE requirement, the authors provide a theoretically sound and practically viable solution for complex control problems. Future work will focus on extending this framework to handle hard constraints on states and control inputs, potentially through integration with Model Predictive Control (MPC).
Summary updated