Introduction
Proportional-Integral-Derivative (PID) control remains the cornerstone of industrial robotics due to its simplicity and reliability. However, tuning PID gains ($k_p, k_i, k_d$) for Multi-Input Multi-Output (MIMO) systems presents a significant challenge. In such systems, control loops are often coupled, meaning adjustments in one parameter can unpredictably affect other outputs. Traditional tuning methods typically rely on precise mathematical models, which are frequently unavailable for complex robots operating in dynamic environments. While Reinforcement Learning (RL) offers a model-free alternative, it often suffers from slow convergence and potential instability during the training phase.
Methodology
The proposed LSAC-PID architecture addresses these challenges by integrating Deep RL with classical control theory. The system consists of two primary layers: a low-level PID layer and a high-level RL layer.
The PID Layer
An incremental PID controller is employed to handle low-level motor commands. Unlike traditional position-based PID, the incremental approach calculates the change in the control signal, leading to smoother transitions and reduced mechanical wear. In the MIMO block, multiple controllers (e.g., for linear and angular velocity) are tuned simultaneously.
The RL Layer (LSAC)
The Soft Actor-Critic (SAC) agent serves as the high-level tuner. It observes the robot's state—derived from camera data and onboard sensors—and dynamically outputs optimal PID parameters. The core innovation of this work is Lyapunov Reward Shaping. By incorporating Lyapunov stability theory into the reward function, the authors penalize unstable transitions and reward the agent for moving toward stable states. This theoretical grounding preserves the optimal policy while significantly accelerating the learning process.
Experimental Setup
The effectiveness of the LSAC-PID approach was evaluated through both simulation and real-world experiments.
Simulation and Real-World Platforms
Simulations were conducted using the Robot Operating System (ROS) and Gazebo, featuring three distinct paths of varying complexity. For real-world validation, a physical mobile robot equipped with mecanum wheels and a camera was used to test the "sim-to-real" transferability of the trained models.
Vision Processing
To support line-following tasks, the researchers implemented an "upward region growing" algorithm. This vision system allows the robot to accurately identify paths and ignore environmental interference or ambiguous "forks" in the road, providing a stable state input for the RL agent.
Results and Discussion
The LSAC-PID algorithm demonstrated superior performance across several metrics:
- Convergence Speed: LSAC-PID reached a stable success rate in 1058 episodes, compared to 1220 episodes for standard SAC-PID.
- Success Rates: In complex simulations, LSAC-PID achieved an 89% success rate, outperforming the 84% of the baseline.
- Robustness: Models trained on complex paths exhibited 100% success when transferred to simpler tracks, indicating the learning of generalized control principles.
- Real-World Performance: The system maintained a 100% success rate on standard real-world paths and remained highly effective (80-90%) on tracks with sharp, complex turns.
Conclusion
The LSAC-PID approach successfully bridges the gap between "black-box" reinforcement learning and "white-box" classical control. By using Lyapunov functions to shape rewards, the system achieves faster, more stable tuning for MIMO robotic systems without requiring an underlying physical model. Future work will focus on generalizing this approach to other complex platforms, such as robotic manipulators and quad-rotors, and addressing the manual selection of RL hyperparameters like the reward scaling factor.
Summary updated