Towards AI-Controlled Movement Restoration: Learning FES-Cycling Stimulation with Reinforcement Learning

Nat Wannawas, A. Aldo FaisalICORR 2023 · DOI

TL;DR

This paper introduces an AI-driven approach to automatically discover optimal functional electrical stimulation (FES) patterns for cycling, a rehabilitation exercise. By combining biomechanical simulations with reinforcement learning and then fine-tuning with real-world data, their method bypasses the need for complex sensors and manual tuning, leading to improved cycling performance.

What Problem Does This Paper Address?

Functional Electrical Stimulation (FES) is a promising rehabilitation technique that uses electrical signals to stimulate muscles, enabling individuals with paralysis to perform exercises like cycling. However, the effectiveness of FES cycling heavily relies on precise stimulation patterns, which are currently determined manually through a time-consuming and challenging trial-and-error process. This manual tuning is not only inefficient but also requires expert knowledge and can be difficult for users to perform independently at home. Existing automated methods often rely on complex setups like EMG sensors or force-measuring pedals, which are not practical for home use. This paper aims to bridge this gap by developing an AI-based method that can automatically learn effective FES cycling patterns without requiring specialized hardware.

Methods

The authors propose a two-phase approach:

  1. Model-Based Phase:
    • Musculoskeletal Modeling: They utilize OpenSim, an open-source biomechanical simulation software, to create a customized musculoskeletal model of the cyclist and their cycling setup. This model captures joint mechanics and muscle dynamics.
  2. Reinforcement Learning (RL): Soft Actor-Critic (SAC), a state-of-the-art RL algorithm, is employed to learn an initial "model-based" stimulation pattern. The RL agent interacts with the OpenSim simulation, learning to control muscle stimulation to achieve efficient cycling. The state space includes crank angle and velocity, and the action space controls stimulation intensities for key leg muscles (quadriceps, hamstrings, gluteus maximus). The reward function is designed to encourage high cadence while penalizing excessive stimulation to promote efficiency.
  3. Fine-Tuning Phase:
    • Offline RL: After obtaining a model-based pattern, real-world cycling data is collected using this pattern. This data is then used to train the RL agent in an "offline" setting. Conservative Q-Learning (CQL) is used to handle the offline learning, allowing the agent to learn from a fixed dataset without further direct interaction with the real system. This phase refines the pattern to better suit the individual and real-world conditions.

This two-phase approach allows for robust initial pattern discovery in simulation and then adaptation to real-world nuances using limited real-world data.

Datasets

The paper describes two main data collection and usage scenarios:

  • Simulation Data: The OpenSim musculoskeletal model is used to generate extensive simulation data for the model-based RL training.
  • Real-World Cycling Data: For the fine-tuning phase, approximately 100 seconds of cycling data were collected from a single healthy subject. This data was generated by performing ten 10-second cycling sessions using slightly adjusted model-based patterns. This data was converted into experience tuples for offline RL training.

The paper also mentions collecting EMG data from a healthy subject during voluntary cycling to generate an EMG-based pattern for comparison.

Key Insights and Ideas

  • AI for Personalized Rehabilitation: This work demonstrates the potential of AI, specifically reinforcement learning, to personalize rehabilitation devices like FES cycling systems. By learning individual-specific stimulation patterns, the system can offer more effective and comfortable therapy.
  • Bridging Simulation and Reality: The two-phase approach effectively leverages the strengths of both simulation (for initial exploration and robustness) and real-world data (for fine-tuning and practical adaptation). This is crucial for developing AI systems that work reliably outside of controlled lab environments.
  • Sensorless Pattern Discovery: A significant contribution is the ability to find effective FES patterns without requiring specialized sensors like EMG or force pedals. This makes the technology more accessible and practical for home-use rehabilitation.
  • Human-in-the-Loop AI: The paper highlights the feasibility and potential of human-in-the-loop AI in real-world rehabilitation, where human interaction and feedback are integral to the learning process.

Key Results

  • Model-Based Robustness: The model-based RL approach was tested across 10 different cycling configurations (seat positions, leg lengths) and in both 2- and 3-muscle cycling settings, demonstrating robustness.
  • Improved Cycling Speed: In real-world experiments, the out-of-the-box model-based pattern achieved an average cycling speed of 49.83 RPM, which was slightly higher than the EMG-based pattern (48.13 RPM).
  • Fine-Tuned Performance: The fine-tuned pattern, derived from offline RL on real-world data, achieved the highest average cycling speed of 52.37 RPM.
  • Efficient Fine-Tuning: The fine-tuning process required only about 100 seconds of cycling data and approximately 1 minute of offline RL training to achieve performance improvements.
  • Pattern Differences: The study observed differences in stimulation timing between EMG-based, model-based, and fine-tuned patterns, with the fine-tuned pattern showing earlier muscle activation, contributing to higher speeds.

Limitations and Future Work

  • Muscular Fatigue: The authors acknowledge that muscular fatigue can affect cadence and influence fine-tuned pattern results. They suggest that this could be addressed by weighting cadence data or using methods to infer hidden states.
  • Paralyzed Individuals: The current study was conducted on a healthy subject. Testing on paralyzed individuals is a crucial next step, which may require assistance for initiating cycling and careful data processing to account for inertia-influenced periods.
  • Model Optimization: The current method does not exploit the collected data to optimize the OpenSim model itself, which could potentially lead to better model-based patterns and a shorter fine-tuning phase.
  • Objective Function: The current reward function is optimized for high-speed cycling. Future work could involve modifying this objective for other cycling purposes (e.g., endurance, muscle strengthening) by adjusting parameters like the action penalty weight.
  • Low Roller Resistance: The experiment was conducted with very low roller resistance, which allowed for immediate observation of stimulation effectiveness. Future work should explore performance under more realistic, higher resistance conditions.
  • Interface Development: Developing low-cost, dedicated interfaces for integrating crank angle measurement devices with existing cycling systems is identified as a future direction to enhance user accessibility.

Summary updated

BibTeX
@inproceedings{wannawas2023towards,
  author = {Nat Wannawas and A. Aldo Faisal},
  title = {{Towards AI-Controlled Movement Restoration: Learning FES-Cycling Stimulation with Reinforcement Learning}},
  booktitle = {ICORR},
  year = {2023},
  doi = {10.1109/icorr58425.2023.10304767},
}
Figure