A Self-adaptive LSAC-PID Approach based on Lyapunov Reward Shaping for Mobile Robots

Xinyi Yu, Siyu Xu, Yuehai Fan, Linlin Ou

Introduction

Proportional-Integral-Derivative (PID) control remains the cornerstone of industrial robotics due to its simplicity and reliability. However, tuning PID gains ($k_p, k_i, k_d$) for Multi-Input Multi-Output (MIMO) systems presents a significant challenge. In such systems, control loops are often coupled, meaning adjustments in one parameter can unpredictably affect other outputs. Traditional tuning methods typically rely on precise mathematical models, which are frequently unavailable for complex robots operating in dynamic environments. While Reinforcement Learning (RL) offers a model-free alternative, it often suffers from slow convergence and potential instability during the training phase.

Methodology

The proposed LSAC-PID architecture addresses these challenges by integrating Deep RL with classical control theory. The system consists of two primary layers: a low-level PID layer and a high-level RL layer.

Detailed RL agent and MIMO PID block structure

The PID Layer

An incremental PID controller is employed to handle low-level motor commands. Unlike traditional position-based PID, the incremental approach calculates the change in the control signal, leading to smoother transitions and reduced mechanical wear. In the MIMO block, multiple controllers (e.g., for linear and angular velocity) are tuned simultaneously.

The RL Layer (LSAC)

The Soft Actor-Critic (SAC) agent serves as the high-level tuner. It observes the robot's state—derived from camera data and onboard sensors—and dynamically outputs optimal PID parameters. The core innovation of this work is Lyapunov Reward Shaping. By incorporating Lyapunov stability theory into the reward function, the authors penalize unstable transitions and reward the agent for moving toward stable states. This theoretical grounding preserves the optimal policy while significantly accelerating the learning process.

Experimental Setup

The effectiveness of the LSAC-PID approach was evaluated through both simulation and real-world experiments.

Simulation vs. real-world robot platforms

Simulation and Real-World Platforms

Simulations were conducted using the Robot Operating System (ROS) and Gazebo, featuring three distinct paths of varying complexity. For real-world validation, a physical mobile robot equipped with mecanum wheels and a camera was used to test the "sim-to-real" transferability of the trained models.

Vision Processing

To support line-following tasks, the researchers implemented an "upward region growing" algorithm. This vision system allows the robot to accurately identify paths and ignore environmental interference or ambiguous "forks" in the road, providing a stable state input for the RL agent.

Results and Discussion

The LSAC-PID algorithm demonstrated superior performance across several metrics:

  • Convergence Speed: LSAC-PID reached a stable success rate in 1058 episodes, compared to 1220 episodes for standard SAC-PID.
  • Success Rates: In complex simulations, LSAC-PID achieved an 89% success rate, outperforming the 84% of the baseline.
  • Robustness: Models trained on complex paths exhibited 100% success when transferred to simpler tracks, indicating the learning of generalized control principles.
  • Real-World Performance: The system maintained a 100% success rate on standard real-world paths and remained highly effective (80-90%) on tracks with sharp, complex turns.

Conclusion

The LSAC-PID approach successfully bridges the gap between "black-box" reinforcement learning and "white-box" classical control. By using Lyapunov functions to shape rewards, the system achieves faster, more stable tuning for MIMO robotic systems without requiring an underlying physical model. Future work will focus on generalizing this approach to other complex platforms, such as robotic manipulators and quad-rotors, and addressing the manual selection of RL hyperparameters like the reward scaling factor.

Summary updated

Abstract

To solve the coupling problem of control loops and the adaptive parameter tuning problem in the multi-input multi-output (MIMO) PID control system, a self-adaptive LSAC-PID algorithm is proposed based on deep reinforcement learning (RL) and Lyapunov-based reward shaping in this paper. For complex and unknown mobile robot control environment, an RL-based MIMO PID hybrid control strategy is firstly presented. According to the dynamic information and environmental feedback of the mobile robot, the RL agent can output the optimal MIMO PID parameters in real time, without knowing mathematical model and decoupling multiple control loops. Then, to improve the convergence speed of RL and the stability of mobile robots, a Lyapunov-based reward shaping soft actor-critic (LSAC) algorithm is proposed based on Lyapunov theory and potential-based reward shaping method. The convergence and optimality of the algorithm are proved in terms of the policy evaluation and improvement step of soft policy iteration. In addition, for line-following robots, the region growing method is improved to adapt to the influence of forks and environmental interference. Through comparison, test and cross-validation, the simulation and real-environment experimental results all show good performance of the proposed LSAC-PID tuning algorithm.
BibTeX
@misc{yu2021self,
  author = {Xinyi Yu and Siyu Xu and Yuehai Fan and Linlin Ou},
  title = {{A Self-adaptive LSAC-PID Approach based on Lyapunov Reward Shaping for Mobile Robots}},
  year = {2021},
  eprint = {2111.02283},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.02283},
}
Figure