Reinforcement Learning Based Fast Self-Recalibrating Decoder for Intracortical Brain–Machine Interface

Peng Zhang, Lianying Chao, Yuting Chen, Xuan Ma, Weihua Wang, Jiping He, Jian Huang, Qiang LiSensors 2020 · DOI

TL;DR

This paper introduces TMAGRL, a novel self-recalibrating decoder for brain-machine interfaces (BMIs) that combines reinforcement learning (RL) with transfer learning (TL). TMAGRL significantly improves decoding accuracy and dramatically speeds up weight updating, addressing the critical challenge of non-stationarity in neural signals for real-world BMI applications.

What Problem Does This Paper Address?

Intracortical brain-machine interfaces (iBMIs) hold immense promise for restoring motor function in individuals with paralysis. A crucial component of these systems is the decoder, which translates neural activity into control signals. However, neural recordings are inherently non-stationary, meaning the relationship between neural signals and intended movements changes over time due to factors like electrode drift or biological responses. This necessitates frequent, often daily, retraining of the decoder using supervised learning, which is cumbersome and impractical for real-world applications, especially in clinical settings. While reinforcement learning (RL) offers a path towards adaptive decoders that can learn from user interaction without explicit labels, quickly adapting to new neural data while maintaining high performance remains a significant challenge. Existing RL-based approaches can be slow to explore new information and prone to instability.

Methods

The core innovation of this paper is the TMAGRL (Transfer Learning and Mini-Batch based Attention-Gated Reinforcement Learning) algorithm. TMAGRL builds upon the Attention-Gated Reinforcement Learning (AGREL) framework but introduces several key enhancements:

  • Transfer Learning (TL) via Domain Adaptation: To bridge the gap between historical and current neural data, TMAGRL employs a Principal Component Analysis-based Domain Adaptation (PDA) method. This unsupervised approach projects both historical and current data into a shared feature space, reducing disparities and improving the initial decoder's ability to adapt. This is a novel application of TL in the context of online RL-based BMIs, where target domain labels are typically unavailable.
  • Mini-Batch (MB) Updates: Instead of using all available historical data for retraining or updating, TMAGRL utilizes mini-batches of recent data. This significantly accelerates the weight updating process and helps mitigate overfitting.
  • Weight Updating Schemes: The paper introduces specific weight updating strategies designed to optimize the learning process and further prevent overfitting. Notably, samples with high prediction accuracy are excluded from weight updates, preventing them from unduly influencing the model and ensuring more robust learning.

Datasets

The study utilized intracortical neural data recorded from two adult male rhesus macaques (Monkey M and Monkey B) performing reaching and grasping tasks. The data was collected over approximately one month, with multiple sessions and datasets per session. For training and testing, the researchers used historical data from previous days to simulate real-world scenarios where a decoder needs to adapt to new, unseen data.

Key Insights and Ideas

  • Bridging the Historical-Current Data Gap: The application of transfer learning, specifically through domain adaptation, is a critical step towards making adaptive BMI decoders practical. By projecting data into a common space, TMAGRL effectively reduces the "curse of dimensionality" and the impact of non-stationarity.
  • Speed is Crucial for Real-World BMIs: The dramatic improvement in weight updating speed (up to 70x faster than AGREL) is a game-changer. This speed is essential for real-time adaptation and makes the decoder viable for wearable devices and clinical use where immediate responsiveness is paramount.
  • Intelligent Weight Updates Prevent Overfitting: The strategy of excluding high-confidence predictions from weight updates is a clever way to prevent the decoder from becoming overly specialized to specific neural patterns that might be transient, thus promoting generalization and robustness.
  • RL's Potential Beyond Supervised Learning: This work demonstrates that RL, when combined with effective adaptation strategies like TL and MB updates, can overcome the limitations of supervised learning in dynamic environments like neural recordings.

Key Results

  • High Decoding Accuracy: TMAGRL achieved approximately 90% classification accuracy across all datasets for both monkeys, outperforming other methods, including a daily retrained decoder.
  • Significant Speedup: The weight updating speed of TMAGRL was approximately 70 times faster than the AGREL method, with online testing taking as little as 5 ms per update compared to AGREL's ~391 ms.
  • Robustness to Time Separation: TMAGRL's decoding performance remained stable even when historical data used for initial training was separated by several days from the current testing data, highlighting its adaptive capabilities.
  • Improved Convergence: TMAGRL showed faster and more stable convergence during initial decoder training compared to AGREL.

Limitations and Future Work

  • Limited Classification: The study focused on three-class classifications. The authors suggest exploring TMAGRL for more complex classifications or continuous neural decoding.
  • Offline Testing: While the study simulated online testing, it was not conducted in a full brain control task. The authors propose testing the integrated method with a wireless wearable sensor system on a brain control task to further verify its clinical applicability.
  • Co-adaptation: The paper acknowledges that in a true brain control task, the brain, decoder, and environment co-adapt. Future work could explore this dynamic.
  • Mini-Batch Optimization: While the MB approach was effective, the authors suggest that prioritized experience replay might offer further improvements.

Summary updated

Abstract

This paper proposed a self-recalibrating decoder which achieved a good and robust decoding performance with fast weight updating and might facilitate its application in wearable device and clinical practice.
BibTeX
@inproceedings{zhang2020reinforcement,
  author = {Peng Zhang and Lianying Chao and Yuting Chen and Xuan Ma and Weihua Wang and Jiping He and Jian Huang and Qiang Li},
  title = {{Reinforcement Learning Based Fast Self-Recalibrating Decoder for Intracortical Brain–Machine Interface}},
  booktitle = {Sensors},
  year = {2020},
  doi = {10.3390/s20195528},
}
Figure