Toward Reliable Designs of Data-Driven Reinforcement Learning Tracking Control for Euler-Lagrange Systems

Zhikai Yao, Jennie Si, Ruofan Wu, Jianyong Yao

Abstract

This paper addresses reinforcement learning based, direct signal tracking control with an objective of developing mathematically suitable and practically useful design approaches. Specifically, we aim to provide reliable and easy to implement designs in order to reach reproducible neural network-based solutions. Our proposed new design takes advantage of two control design frameworks: a reinforcement learning based, data-driven approach to provide the needed adaptation and (sub)optimality, and a backstepping based approach to provide closed-loop system stability framework. We develop this work based on an established direct heuristic dynamic programming (dHDP) learning paradigm to perform online learning and adaptation and a backstepping design for a class of important nonlinear dynamics described as Euler-Lagrange systems. We provide a theoretical guarantee for the stability of the overall dynamic system, weight convergence of the approximating nonlinear neural networks, and the Bellman (sub)optimality of the resulted control policy. We use simulations to demonstrate significantly improved design performance of the proposed approach over the original dHDP.

Introduction

Euler-Lagrange (EL) systems, which encompass a wide range of mechanical systems including robotic manipulators, underwater vehicles, and satellite antennas, are fundamental to modern engineering. Achieving precise tracking control in these systems typically requires an accurate mathematical model. However, real-world dynamics are often too complex or uncertain to model perfectly.

While data-driven Reinforcement Learning (RL) offers a promising path to learning these dynamics from experience, it faces significant hurdles in industrial deployment. Specifically, RL agents often lack inherent stability guarantees, suffer from low reproducibility due to sensitivity to initial conditions, and frequently rely on intermediate reference models that introduce additional approximation errors. This research proposes a "reliable design" that is both mathematically stable and practically reproducible.

Proposed Methodology

The innovation lies in the integration of Backstepping Control and direct Heuristic Dynamic Programming (dHDP). This hybrid framework uses backstepping to provide a structural "safety rail," ensuring the system remains stable while the RL agent explores and learns the optimal policy.

The proposed hybrid control architecture (a) and the internal structure of the dHDP module (b).

As illustrated in the block diagram above, the system operates as follows:

  • Backstepping for Stability: A discrete-time backstepping approach provides the baseline feedback structure, ensuring the closed-loop system remains stable.
  • dHDP for Optimality: The dHDP module consists of an "Actor" network (ANN) that learns the optimal control policy and a "Critic" network (CNN) that approximates the cost-to-go (Bellman equation).
  • Feed-forward Compensation: Instead of the RL agent managing the entire control task, it is specifically tasked with providing feed-forward compensation to account for unknown system dynamics ($f(k)$).
  • Direct Trajectory Tracking: This design allows the system to track a desired trajectory directly, eliminating the need for an intermediate reference model and reducing cumulative error.

By using backstepping to narrow the search space for the neural networks, the authors significantly increase the likelihood of finding a (sub)optimal solution compared to vanilla RL methods.

Experimental Results and Discussion

The proposed method was validated using high-fidelity simulations of single-link and two-link robot manipulators. The performance was compared against a standard dHDP approach and traditional feedback stabilizing controllers across various scenarios involving external disturbances and noise.

Key Findings:

  • Success Rate: In a 50-trial test on a single-link robot, the vanilla dHDP approach succeeded only 14% of the time. In contrast, the proposed hybrid method achieved a 100% success rate.
  • Precision: The Mean Square Error (MSE) for the hybrid method was approximately four orders of magnitude lower than the successful vanilla dHDP cases ($2.135 \times 10^{-5}$ vs $1.454 \times 10^{-1}$).
  • Robustness: The system maintained high performance even when subjected to unmodeled disturbances, such as sudden pulses or Gaussian noise, significantly outperforming traditional feedback linearization.

Theoretical analysis further confirmed that the tracking errors and neural network weights are Uniformly Ultimately Bounded (UUB), providing a rigorous guarantee that the system will remain within a predictable range of the target trajectory.

Conclusion

This research demonstrates that combining the structural stability of backstepping with the adaptive capabilities of RL creates a more reliable and reproducible control framework for Euler-Lagrange systems. While the current results are simulation-based and require hyperparameter tuning, they provide a strong mathematical foundation for future implementation on physical robotic platforms. Future work will focus on extending this framework to broader classes of nonlinear systems and addressing real-world hardware constraints such as sensor latency.

Summary updated

Abstract

This paper addresses reinforcement learning based, direct signal tracking control with an objective of developing mathematically suitable and practically useful design approaches. Specifically, we aim to provide reliable and easy to implement designs in order to reach reproducible neural network-based solutions. Our proposed new design takes advantage of two control design frameworks: a reinforcement learning based, data-driven approach to provide the needed adaptation and (sub)optimality, and a backstepping based approach to provide closed-loop system stability framework. We develop this work based on an established direct heuristic dynamic programming (dHDP) learning paradigm to perform online learning and adaptation and a backstepping design for a class of important nonlinear dynamics described as Euler-Lagrange systems. We provide a theoretical guarantee for the stability of the overall dynamic system, weight convergence of the approximating nonlinear neural networks, and the Bellman (sub)optimality of the resulted control policy. We use simulations to demonstrate significantly improved design performance of the proposed approach over the original dHDP.
BibTeX
@misc{yao2020toward,
  author = {Zhikai Yao and Jennie Si and Ruofan Wu and Jianyong Yao},
  title = {{Toward Reliable Designs of Data-Driven Reinforcement Learning Tracking Control for Euler-Lagrange Systems}},
  year = {2020},
  eprint = {2101.00068},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2101.00068},
}
Figure