Self-Learning Variable Structure Control for a Class of Sensor-Actuator Systems

Sanfeng Chen, Shuai Li, Bo Liu, Yuesheng Lou, Yongsheng Liang

TL;DR

This paper introduces a novel "self-learning" variable structure control method for sensor-actuator systems that, unlike traditional approaches, does not require prior knowledge of the system's mathematical model. By formulating the problem from an optimal control perspective and employing iterative methods inspired by adaptive dynamic programming, the authors demonstrate a model-free approach that effectively stabilizes complex systems.

What Problem Does This Paper Address?

Many advanced automatic systems rely on precise control of sensor-actuator systems. A common modeling framework for these systems is the Euler-Lagrange equation. However, designing effective control strategies, particularly variable structure control (VSC), for these systems often necessitates accurate knowledge of their mathematical model, including parameters like inertial matrices and friction coefficients. This model dependency makes VSC brittle, as performance degrades significantly with model errors. The challenge this paper addresses is the development of a robust VSC strategy that can achieve stable control without requiring this detailed, often hard-to-obtain, model information. The goal is to create a controller that can learn and adapt online, using only the system's input and output data.

Methods

The core of the proposed method lies in a "self-learning" variable structure control strategy that leverages optimal control principles and iterative learning.

  1. Optimal Control Formulation: The problem of stabilizing the sensor-actuator system is framed as an infinite-horizon optimal control problem. The objective is to find a control input sequence that minimizes a defined cost function, which includes a utility function that penalizes deviations from a desired sliding surface.
  2. Principle of Optimality and Bellman Equation: The authors apply the principle of optimality to derive a Bellman equation that characterizes the optimal cost function. This equation forms the basis for an iterative approach to finding the optimal control law.
  3. Iterative Solution: An iterative procedure is proposed to approximate the optimal cost function and, consequently, the optimal control law. This involves repeatedly applying a Bellman operator.
  4. Adaptive Dynamic Programming (ADP) / Reinforcement Learning: To make the iterative solution practical for real-time implementation, the paper employs a learning-based strategy inspired by adaptive dynamic programming and reinforcement learning. This involves training two neural network models:
  5. Critic Model: Learns to approximate the optimal cost function (J*).
  6. Action Model: Learns to approximate the optimal control action (u*). These models are updated iteratively using back-propagation rules based on defined error signals for both the critic and action models. This allows the system to "learn" the optimal control policy over time, eliminating the need for repeated online optimization after a sufficient learning period.

Datasets

Not discussed in the paper. The method is demonstrated using a simulated cart-pole system, which serves as a benchmark for evaluating the control strategy.

Key Insights and Ideas

  • Model-Free VSC is Achievable: The most significant insight is the demonstration that variable structure control, known for its robustness, can be made model-free through intelligent learning techniques. This opens up VSC to a wider range of applications where precise system models are unavailable or difficult to obtain.
  • Bridging Optimal Control and Learning: The paper effectively bridges the gap between theoretical optimal control (Bellman equations) and practical, adaptive control through the use of parameterized critic and action models. This allows for online learning of complex control policies.
  • Self-Learning for Robustness: By learning the control policy online, the system can adapt to unmodeled dynamics and parameter variations, leading to enhanced robustness compared to traditional model-based VSC.
  • Efficiency through Learned Models: After a learning phase, the control action can be directly computed by the learned action model, bypassing computationally intensive online optimization, which is crucial for real-time applications.

Key Results

The proposed self-learning variable structure control method was applied to the stabilization of a cart-pole system. The simulations demonstrated:

  • Effective Stabilization: The controller successfully stabilized the cart-pole system to the inverted position (pendulum upright), achieving the control objective.
  • Small Steady-State Error: For the pendulum angle ($\theta$), the system achieved stabilization within a small vicinity around zero, with an error of approximately $\pm 0.1$ rads.
  • Model-Free Performance: The method achieved these results without requiring prior knowledge of the cart-pole system's parameters (e.g., mass, friction, length).
  • Efficiency: The learned models allowed for efficient control execution after the initial learning period.

Limitations and Future Work

  • Theoretical Proof of Convergence: The authors acknowledge that while simulations demonstrate effectiveness, a rigorous theoretical proof of the convergence of the iterative learning procedure and the stability of the closed-loop system is a direction for future work.
  • Performance Limits: Exploring the performance limits of the proposed strategy is also identified as a future research avenue.
  • Generalization to Other Systems: The paper suggests extending the application of this method to other mechanical systems modeled by Euler-Lagrange equations, such as manipulators, which would be a valuable next step.
  • Parameter Tuning: The performance of the learning process is dependent on the choice of step sizes ($l_c(n)$, $l_a(n)$) and the discount factor ($\gamma$), which are typically tuned empirically. Further research into adaptive tuning of these parameters could be beneficial.

Summary updated

Abstract

Variable structure strategy is widely used for the control of sensor-actuator systems modeled by Euler-Lagrange equations. However, accurate knowledge on the model structure and model parameters are often required for the control design. In this paper, we consider model-free variable structure control of a class of sensor-actuator systems, where only the online input and output of the system are available while the mathematic model of the system is unknown. The problem is formulated from an optimal control perspective and the implicit form of the control law are analytically obtained by using the principle of optimality. The control law and the optimal cost function are explicitly solved iteratively. Simulations demonstrate the effectiveness and the efficiency of the proposed method.
BibTeX
@inproceedings{chen2012self,
  author = {Sanfeng Chen and Shuai Li and Bo Liu and Yuesheng Lou and Yongsheng Liang},
  title = {{Self-Learning Variable Structure Control for a Class of Sensor-Actuator Systems}},
  booktitle = {Sensors},
  year = {2012},
  url = {https://openreview.net/forum?id=4mhYnsPj68nS},
}
Figure