Abstract
This study introduces a novel approach to robot-assisted ankle rehabilitation by proposing a Dual-Agent Multiple Model Reinforcement Learning (DAMMRL) framework, leveraging multiple model adaptive control (MMAC) and co-adaptive control strategies. In robot-assisted rehabilitation, one of the key challenges is modelling human behaviour due to the complexity of human cognition and physiological systems. Traditional single-model approaches often fail to capture the dynamics of human-machine interactions. Our research employs a multiple model strategy, using simple sub-models to approximate complex human responses during rehabilitation tasks, tailored to varying levels of patient incapacity. The proposed system's versatility is demonstrated in real experiments and simulated environments. Feasibility and potential were evaluated with 13 healthy young subjects, yielding promising results that affirm the anticipated benefits of the approach. This study not only introduces a new paradigm for robot-assisted ankle rehabilitation but also opens the way for future research in adaptive, patient-centred therapeutic interventions.
Introduction
Robot-assisted rehabilitation is a high-stakes game of "follow the leader," but determining who should lead is a significant challenge. Traditional approaches often use a single, static model to represent human behavior. However, human neurological and physiological responses are nonlinear, time-varying, and highly individual. Modeling this complexity accurately usually leads to the "curse of dimensionality," where models become too computationally heavy for real-time use. Furthermore, in a clinical setting, a robot must adapt to a patient’s changing fatigue levels and recovery stages, while the patient simultaneously learns how to interact with the robot. This "co-adaptation" is notoriously difficult to supervise and optimize using standard single-agent reinforcement learning (RL).
Methodology: The DAMMRL Framework
The authors propose Dual-Agent Multiple Model Reinforcement Learning (DAMMRL) to address these challenges. The core innovation lies in three key areas:
- Dual-Agent MDP: The system is modeled as a Cooperative Adaptive Markov Decision Process (CAMDP). One agent is the Machine (Agent 1), which controls motor torque, and the other is a Virtual Human (Agent 0), which represents the patient's cognitive and musculoskeletal response.
- Multiple Model Strategy (MMAC): Instead of one massive neural network trying to learn every possible human reaction, the framework uses a library of simple sub-models (such as PD and PID controllers with different parameters). The RL agent’s "action" is the decision of which sub-model to switch to, simplifying the action space and improving explainability.
- PPO Algorithm: Proximal Policy Optimization (PPO), an actor-critic RL algorithm, is used to train the switching logic. The "Actor" decides the control strategy, while the "Critic" evaluates how well that strategy meets rehabilitation goals.
Experimental Design and Biofeedback Loop
To bridge the gap between simulation and real-world application, the system uses a high-fidelity model of the ankle robot built in Simulink/SimMechanics for pre-training. Real-world evaluation was conducted with 13 healthy young subjects at the Jinan Key Lab of Intelligent Rehabilitation Robotics.
A critical component is the Human-Machine Interface (HMI) & Biofeedback loop. The system displays digits (from -2 to 2) to the user, acting as a stimulus to prompt more or less force on the pedal. The resulting pressure is measured via sensors, creating a measurable "input-output" loop that treats the human's complex nervous system as a "black box" responding to visual stimuli with physical torque.
Key Results and Analysis
The system was tested under various settings prioritizing different therapeutic goals. The authors designed a composite reward function balancing Tracking Error (staying on path), Comfort (minimizing jerk), and Effort (ensuring patient participation).
- Active vs. Passive Performance: Settings encouraging active participation achieved significantly higher Value Functions and lower Tracking Errors (MSE as low as 4.57) compared to less optimized settings. In passive modes, the robot precisely guided the limb with MSE as low as 1.46.
- Personalization: The system successfully adapted to individual variances, such as subjects with stronger biofeedback reactions, demonstrating the framework's robustness.
- Consistency: Experimental results with real subjects closely matched the simulated training results, validating the multiple model approach.
Conclusion and Future Work
While promising, the DAMMRL framework requires further validation with stroke patients or those with severe motor impairments, whose biofeedback loops may be more unpredictable. Future research will explore neural network pruning for more efficient switching logic on low-power devices and the integration of expanded biofeedback data, such as EEG (brainwaves) or image-based emotion classification, to better capture the patient's internal state during therapy.
Summary updated