Dual regularized policy updating and shiftpoint detection for automated deployment of reinforcement learning controllers on industrial mechatronic systems

Victor Vantilborgh, Tom Staessens, Wannes De Groote, Guillaume CrevecoeurControl 2023 · DOI

Introduction

Reinforcement Learning (RL) has emerged as a powerful tool for controlling complex mechatronic systems, offering the ability to learn directly from data without relying on simplified mathematical models. Despite this potential, RL agents often exhibit significant "brittleness" in industrial settings. When physical environments change—due to mechanical wear, increased friction, or motor degradation—agent performance typically degrades, potentially leading to unsafe operations.

To address these challenges, this paper introduces an automated pipeline designed to detect distributional shifts in system behavior and update control policies safely and efficiently. The approach combines robust model-based shiftpoint detection with a novel teacher-student dual-regularization architecture.

Methodology

The proposed framework automates the lifecycle of an RL controller through a continuous feedback loop involving the physical system, a detection module, and a policy updating mechanism.

Figure 1: Overall system architecture illustrating the integration of shiftpoint detection and the dual-controller policy updating loop.

Shiftpoint Detection

The first stage of the pipeline involves identifying when the system has changed enough to warrant a policy update. Three detection strategies were benchmarked:

  1. Statistical Metrics: Utilizing tests such as the Kolmogorov-Smirnov test to compare state-action data distributions.
  2. Uncertainty-based Monitoring: Tracking the epistemic uncertainty of the RL agent via Monte Carlo Dropout or Bootstrapping.
  3. Model-based Estimation: Training a neural network as a state predictor. A significant spike in prediction error serves as a robust indicator of a system shift.

Results indicated that the integrated state predictor was the most effective, often detecting subtle friction changes before performance degradation became apparent.

Dual Regularized Policy Updating

Upon detection of a shift, the system triggers a policy update using a Teacher-Student architecture. This method avoids the pitfalls of retraining from scratch, such as "catastrophic forgetting" or initial instability.

Figure 2: The Teacher-Student architecture utilizing dual regularization (weight and feature consistency) to stabilize policy updates.

The architecture employs two specific regularization techniques:

  • Weight Regularization (CTF): This penalizes the student policy if its neural network weights deviate significantly from a linear transformation of the teacher’s weights.
  • Feature Regularization (CMMD): By applying Maximum Mean Discrepancy, the system ensures that the internal representations (features) learned by the student remain consistent with the teacher's established logic.

Experimental Validation

The pipeline was evaluated using two mechatronic benchmarks:

  • Acrobot: A simulated double pendulum system for testing swing-up and balancing tasks.
  • Slider-Crank Mechanism: Evaluated in both simulation and on physical hardware. The hardware setup utilized a dSPACE 1104 board, with system aging emulated through manual increases in viscous friction.

Key Results and Discussion

The automated pipeline demonstrated significant advantages over standard retraining methods:

  • Rapid Adaptation: In physical slider-crank experiments, the system detected friction shifts and retrained the agent to expert performance levels within only 40 episodes.
  • Enhanced Stability: The dual-regularized approach maintained higher performance levels during the update process compared to "naive" retraining, which often suffers from a performance "dip" at the start of learning.
  • Industrial Feasibility: The architecture allows for heavy training computations to be performed on a connected PC while the active controller runs in real-time on edge hardware.

Conclusion

This research provides a robust framework for the automated maintenance of RL controllers in industrial environments. By integrating proactive shiftpoint detection with a stabilized teacher-student update mechanism, the system ensures that mechatronic controllers can adapt to hardware degradation without sacrificing safety or long-term performance. Future work will explore the pipeline's effectiveness against structural shifts and the potential for fleet-wide learning across multiple identical machines.

Summary updated

BibTeX
@inproceedings{vantilborgh2023dual,
  author = {Victor Vantilborgh and Tom Staessens and Wannes De Groote and Guillaume Crevecoeur},
  title = {{Dual regularized policy updating and shiftpoint detection for automated deployment of reinforcement learning controllers on industrial mechatronic systems}},
  booktitle = {Control},
  year = {2023},
  doi = {10.1016/j.conengprac.2023.105783},
}
Figure