Cross-User Adaptation for sEMG Gesture Recognition via Continual Learning

Aran Ramakrishnan, Shubhi Mishra, Tyler Ham, Assel Kembay, Jason EshraghianarXiv 2026 · arXiv

TL;DR

This paper demonstrates that experience replay is the most effective strategy for allowing prosthetic gesture recognition systems to adapt to new users and new gestures without "forgetting" what they previously learned. The researchers also uncover a critical training pitfall: standard recurrent neural network practices, like carrying hidden states across temporal windows, actually cause model collapse in sEMG data unless the data is shuffled and the state is reset.

What Problem Does This Paper Address?

Surface electromyography (sEMG) uses sensors on the skin to detect muscle activity, which is then decoded into hand gestures for prosthetics. However, sEMG signals are notoriously "messy" and person-specific—factors like muscle anatomy and electrode placement mean a model trained for one person rarely works perfectly for another.

To be truly useful, these systems need to adapt after deployment:

  1. Domain-Incremental Learning (DIL): Adapting to a new user.
  2. Class-Incremental Learning (CIL): Learning new gestures (e.g., adding a "pinch" gesture to an existing set).

The challenge is catastrophic forgetting, where learning a new task causes the model to abruptly lose its ability to perform previous ones. While Continual Learning (CL) aims to solve this, most research has focused on simple feedforward models, leaving the recurrent neural networks (RNNs) typically used for time-series sEMG data largely unstudied in this context.

Methods

The authors compared three CL strategies across two different recurrent architectures:

  • Architectures: The classic LSTM and the HGRN (Hierarchical Gated Recurrent Network), a modern state-space model designed for efficient sequence processing.
  • CL Strategies:
    • Naive Fine-tuning: Simply training on new data (the baseline).
  • Elastic Weight Consolidation (EWC): A regularization method that penalizes changing "important" weights from previous tasks.
  • Experience Replay: Keeping a small buffer of old data and "rehearsing" it alongside new data.
  • Feature Engineering: They compared raw sEMG signals against hand-crafted Hudgins time-domain (TD) features (like Mean Absolute Value and Root Mean Square).

Datasets

The study used the NinaPro Database 1 (DB1), a benchmark dataset containing sEMG data from 27 subjects performing up to 52 different finger and wrist movements.

Key Insights and Ideas

  • The "Stateful" Trap: One of the most surprising findings was that training RNNs in a "stateful" manner—carrying the hidden state from one window to the next to simulate a continuous stream—actually caused the models to fail. Because sEMG data is often dominated by "rest" periods, the models collapsed into predicting only the majority class.
  • Shuffling is Essential: To achieve stable learning, the authors found they had to treat windows as independent samples, shuffling them and resetting the network's state between batches. This "stateless" approach, combined with shuffling, prevented the model from getting stuck in a local minimum.
A 2x2 ablation study showing that shuffling and stateless training are required to maintain "plasticity" (the ability to keep learning).
  • Method Trumps Architecture: Interestingly, the choice of CL strategy (e.g., using Replay vs. EWC) had a much larger impact on performance than the choice of underlying model (LSTM vs. HGRN).
  • Features Need Replay: While time-domain features significantly boosted initial accuracy (by ~40%), this gain was only preserved during class-incremental learning if the model used experience replay. Without it, the "better" features actually led to faster forgetting.

Key Results

  • Superior Accuracy: Experience replay was the clear winner, reaching 83.3% final accuracy in class-incremental scenarios.
  • Minimal Forgetting: Replay limited accuracy drops on old tasks to just 1.8%–5.8%, whereas EWC and Naive fine-tuning suffered massive drops of 16.2%–28.5%.
Task-level accuracy for replay. The "Immediate" and "Final" accuracies stay close together, indicating the model isn't forgetting old subjects or gestures.
  • Efficiency Gains: The modern HGRN architecture achieved comparable accuracy to the LSTM but at 1.6x lower computational cost (FLOPs), making it a strong candidate for low-power prosthetic hardware.
Comparison of CL methods. Replay consistently outperforms EWC and Naive across both DIL and CIL scenarios.

Limitations and Future Work

The authors acknowledge that this study was conducted offline using a pre-recorded dataset. In a real-world prosthetic, data arrives in a true temporal stream, and adaptation must happen in real-time. Future work should explore how these "stateless" shuffling requirements translate to on-device, online learning where data cannot be easily shuffled across long time scales. Additionally, testing on more diverse datasets beyond NinaPro DB1 is needed to ensure these findings generalize across different sensor types and environments.

Summary updated

Abstract

Prosthetic and assistive interfaces increasingly depend on decoding hand gestures from surface electromyography (sEMG). Because the signal is shaped by person-specific factors, the recurrent neural networks (RNNs) commonly used to model these temporal streams must adapt to new wearers and new gesture classes after deployment. Continual learning (CL) enables this adaptation without retraining from scratch, yet applying it to recurrent sEMG models is not straightforward. Standard recurrent training practices can destabilize learning, and which CL strategy holds up under realistic cross-user drift is unestablished. We evaluate three representative CL strategies under domain- and class-incremental drift on NinaPro DB1, and find experience replay to outperform elastic weight consolidation (EWC) and naive fine-tuning. It reaches 83.3% final accuracy for class-incremental drift while limiting forgetting to 1.8-5.8% accuracy drop, as against 16.2-28.5% for EWC and naive training. We further show these models must be trained on shuffled, independently-windowed segments. Streaming windows in temporal order collapses the network into predicting a single dominant gesture, while a shuffled, stateless regime trains stably. Finally, hand-crafted time-domain features raise per-task accuracy by up to about 40% over raw signals, but only replay preserves that gain under class-incremental drift.
BibTeX
@misc{ramakrishnan2026cross,
  author = {Aran Ramakrishnan and Shubhi Mishra and Tyler Ham and Assel Kembay and Jason Eshraghian},
  title = {{Cross-User Adaptation for sEMG Gesture Recognition via Continual Learning}},
  year = {2026},
  eprint = {2610.34011},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.34011},
}
Figure