Model-free tracking control of complex dynamical trajectories with machine learning

Zheng-Meng Zhai, Mohammadamin Moradi, Ling-Wei Kong, Bryan Glaz, Mulugeta Haile, Ying‐Cheng Lai

TL;DR

This paper introduces a novel model-free machine learning framework for robotic tracking control that can follow complex, even chaotic, trajectories using only partial state observations. By leveraging reservoir computing and a unique training approach with stochastic inputs, the system learns to predict and generate control signals without needing an explicit model of the robot's dynamics, making it highly adaptable and robust.

What Problem Does This Paper Address?

Traditionally, controlling a dynamical system to follow a specific trajectory (tracking control) requires a precise mathematical model of the system. This is a significant hurdle in robotics, especially for complex, nonlinear systems where obtaining an accurate model is difficult or impossible. Existing model-free approaches often struggle with high-dimensional states, strong nonlinearities, or require extensive online learning, which can lead to instability. This paper tackles the challenge of designing a robust, model-free tracking controller that can handle arbitrary, complex trajectories, even when only partial information about the system's state is available.

Methods

The core of the proposed solution is a reservoir computing framework, a type of recurrent neural network known for its efficiency in processing time-series data and modeling complex dynamics without extensive training. The key innovations lie in:

  • Model-Free Approach: The controller learns to map observed states to control actions without needing an explicit mathematical model of the robot's dynamics.
  • Partial State Observation: The system only requires a subset of the robot's state variables (e.g., end-effector position and joint angles) to function, making it more practical for real-world applications where full state sensing might be infeasible.
  • Unique Training Strategy:
    • Input Structure: During training, the reservoir computer receives two inputs: the current partial observation y(t) and its immediate future y(t+dt). This teaches the network to associate the first input with the current state and the second with the desired next state.
  • Stochastic Input: A random, stochastic control signal is applied to the robot during training. This generates a "random walk" trajectory, exposing the controller to a wide range of system behaviors and ensuring it learns a general mapping rather than a specific trajectory.
  • Testing Phase Adaptation: In the testing (deployment) phase, the y(t+dt) input is replaced by the desired observation ya(t) from the reference trajectory. The trained network, having learned to associate its input with future states, now generates control signals to make the actual system state y(t+dt) match the desired ya(t). This effectively creates an "inverse model" that dictates the necessary control actions.

Datasets

The paper does not explicitly mention using pre-existing datasets for training. Instead, it generates its own training data by controlling a simulated two-arm robotic manipulator. The evaluation uses a variety of reference trajectories, including:

  • Chaotic Trajectories: Lorenz and Mackey-Glass systems.
  • Periodic Trajectories: A circle and an "eight" figure.

These trajectories are generated and used as benchmarks to test the controller's ability to track complex and ordered paths.

Key Insights and Ideas

  • "Learning the Future" with Partial Data: The core insight is that by training the reservoir computer to predict the immediate future state y(t+dt) based on the current state y(t) and a control signal, it implicitly learns the system's dynamics. This learned mapping can then be repurposed in the testing phase to achieve tracking by feeding it the desired future state ya(t).
  • Stochastic Training for Generality: Using a random-walk trajectory generated by stochastic control signals during training is crucial. It allows the controller to learn a general mapping that is not tied to any specific trajectory, making it adaptable to a wide range of future tasks.
  • Robustness through Data-Driven Learning: The model-free, data-driven nature of the approach inherently imbues the controller with robustness against noise, disturbances, and parameter uncertainties, as it learns from observed behavior rather than relying on potentially inaccurate model parameters.
  • Partial Observation is Sufficient: The success with only partial state information is a significant practical advantage, reducing the complexity and cost of sensor requirements.

Key Results

  • Successful Tracking: The proposed framework successfully tracks a variety of periodic and chaotic reference trajectories with high accuracy.
  • Robustness Demonstrated: The controller shows significant robustness against:
  • Disturbance: Near-zero RMSE (Root Mean Square Error) for disturbance amplitudes up to approximately 10^0.5.
  • Measurement Noise: Tolerance to noise amplitudes up to approximately 10^-1.
  • Parameter Uncertainties: The controller maintains good performance even when arm lengths deviate from their trained values, with performance degrading only when secondary arm length significantly exceeds the primary.
  • Generalizability: The controller, trained on a random walk, can track trajectories it has never seen before, highlighting its adaptability.

Limitations and Future Work

  • Training Complexity: While the trained controller is efficient, the initial training phase with stochastic inputs can be computationally demanding.
  • "Safe Region" for Initial Conditions: The paper notes that selecting the initial end-effector position wisely can improve tracking success, suggesting the concept of a "safe region" for initial conditions. Further investigation into this is suggested.
  • Comparison with Traditional Methods: The authors acknowledge that direct comparison with traditional methods like PID or MPC is difficult because their approach is designed for arbitrary trajectories after a single training session, unlike traditional methods that require parameter tuning for each specific trajectory.
  • Future Work: The paper suggests exploring the effects of stochastic signal amplitude and reference trajectory "speed limits" on tracking performance, and mentions that feed-forward neural networks can also be used as controllers (though with higher computational cost). The authors anticipate deployment in real-world applications like drones and soft robotics.

Summary updated

Abstract

Nonlinear tracking control enabling a dynamical system to track a desired trajectory is fundamental to robotics, serving a wide range of civil and defense applications. In control engineering, designing tracking control requires complete knowledge of the system model and equations. We develop a model-free, machine-learning framework to control a two-arm robotic manipulator using only partially observed states, where the controller is realized by reservoir computing. Stochastic input is exploited for training, which consists of the observed partial state vector as the first and its immediate future as the second component so that the neural machine regards the latter as the future state of the former. In the testing (deployment) phase, the immediate-future component is replaced by the desired observational vector from the reference trajectory. We demonstrate the effectiveness of the control framework using a variety of periodic and chaotic signals, and establish its robustness against measurement noise, disturbances, and uncertainties.
BibTeX
@inproceedings{zhai2023model,
  author = {Zheng-Meng Zhai and Mohammadamin Moradi and Ling-Wei Kong and Bryan Glaz and Mulugeta Haile and Ying‐Cheng Lai},
  title = {{Model-free tracking control of complex dynamical trajectories with machine learning}},
  booktitle = {Nature Communications},
  year = {2023},
  doi = {10.1038/s41467-023-41379-3},
}
Figure