Dynamic Gesture Recognition Using Surface EMG Signals Based on Multi-Stream Residual Network

Zhiwen Yang, Du Jiang, Ying Sun, Bo Tao, Xiliang Tong, Guozhang Jiang, Manman Xu, Juntong Yun, Ying Liu, Baojia Chen, Jianyi KongFrontiers in Bioengineering and Biotechnology 2021 · DOI

Abstract

Gesture recognition technology is widely used in the flexible and precise control of manipulators in the assisted medical field. Our MResLSTM algorithm can effectively perform dynamic gesture recognition. The result of surface EMG signal decoding is applied to the controller, which can improve the fluency of artificial hand control. Much current gesture recognition research using sEMG has focused on static gestures. In addition, the accuracy of recognition depends on the extraction and selection of features. However, static gesture research cannot meet the requirements of natural human-computer interaction and dexterous control of manipulators. Therefore, a multi-stream residual network (MResLSTM) is proposed for dynamic hand movement recognition. This study aims to improve the accuracy and stability of dynamic gesture recognition. Simultaneously, it can also advance the research on the smooth control of the Manipulator. We combine the residual model and the convolutional short-term memory model into a unified framework. The architecture extracts spatiotemporal features from two aspects: global and deep, and combines feature fusion to retain essential information. The strategy of pointwise group convolution and channel shuffle is used to reduce the number of network calculations. A dataset is constructed containing six dynamic gestures for model training. The experimental results show that on the same recognition model, the gesture recognition effect of fusion of sEMG signal and acceleration signal is better than that of only using sEMG signal. The proposed approach obtains competitive performance on our dataset with the recognition accuracies of 93.52%, achieving state-of-the-art performance with 89.65% precision on the Ninapro DB1 dataset. Our bionic calculation method is applied to the controller, which can realize the continuity of human-computer interaction and the flexibility of manipulator control.

Introduction

Real-world human-computer interaction requires the recognition of dynamic gestures—continuous movements like turning a knob or flipping a hand. Most current surface electromyography (sEMG) research, however, focuses on static gestures (e.g., holding a fist). Recognizing dynamic movements is challenging because sEMG signals are noisy, non-stationary, and vary significantly with arm position and muscle fatigue. Traditional manual feature engineering often fails to capture the complex spatiotemporal relationships inherent in muscle activity. To address this, we propose a lightweight, end-to-end deep learning solution capable of handling temporal flow while remaining efficient for real-time applications.

Methods

The proposed MResLSTM (Multi-Stream Residual Long Short-Term Memory) network integrates spatial and temporal feature extraction into a unified framework. The system pipeline involves signal collection, processing into sEMG "images," and classification via the neural network to control a manipulator.

Overview of the dynamic gesture recognition and manipulator control pipeline.

The architecture consists of three main components:

  1. IMResNet (Improved Residual Network): This stage extracts spatial features from sEMG and acceleration data. It utilizes Squeeze-and-Excitation (SE) units for channel attention, allowing the network to prioritize the most relevant muscle sensors.
  2. Variant ConvLSTM (VConvLSTM): To capture temporal dynamics without excessive computational cost, we replaced standard convolutional operations in the LSTM gates with Global Average Pooling (GP) and fully connected layers.
  3. Lightweight Design: Inspired by ShuffleNet, the model employs pointwise group convolutions and channel shuffling to maintain information flow across neuron groups while minimizing parameters.

Results and Discussion

The model was evaluated using a custom dataset of six dynamic gestures and the Ninapro DB1 benchmark (52 gestures).

Key findings include:

  • The Power of Fusion: Combining sEMG with acceleration (ACC) data significantly improves performance by providing a spatial reference that compensates for muscle signal noise and drift.
  • Superior Accuracy: MResLSTM achieved 93.52% accuracy on the custom dataset and a state-of-the-art 89.31% accuracy on Ninapro DB1, outperforming established models like GoogleNet (81.27%) and MSCNN (85.00%).
  • Stability: The multi-stream approach converges faster and demonstrates greater robustness against individual physical differences compared to single-stream models.

Conclusion

MResLSTM provides a robust framework for dynamic gesture recognition, enabling more fluid control of bionic manipulators. Future work will focus on making the model "speed-invariant" to handle varying movement velocities and reducing the accuracy gap caused by subject-specific anatomical differences. We also plan to explore high-density sEMG arrays to further enhance dexterity in human-computer interaction.

Summary updated

Abstract

Gesture recognition technology is widely used in the flexible and precise control of manipulators in the assisted medical field. Our MResLSTM algorithm can effectively perform dynamic gesture recognition. The result of surface EMG signal decoding is applied to the controller, which can improve the fluency of artificial hand control. Much current gesture recognition research using sEMG has focused on static gestures. In addition, the accuracy of recognition depends on the extraction and selection of features. However, Static gesture research cannot meet the requirements of natural human-computer interaction and dexterous control of manipulators. Therefore, a multi-stream residual network (MResLSTM) is proposed for dynamic hand movement recognition. This study aims to improve the accuracy and stability of dynamic gesture recognition. Simultaneously, it can also advance the research on the smooth control of the Manipulator. We combine the residual model and the convolutional short-term memory model into a unified framework. The architecture extracts spatiotemporal features from two aspects: global and deep, and combines feature fusion to retain essential information. The strategy of pointwise group convolution and channel shuffle is used to reduce the number of network calculations. A dataset is constructed containing six dynamic gestures for model training. The experimental results show that on the same recognition model, the gesture recognition effect of fusion of sEMG signal and acceleration signal is better than that of only using sEMG signal. The proposed approach obtains competitive performance on our dataset with the recognition accuracies of 93.52%, achieving state-of-the-art performance with 89.65% precision on the Ninapro DB1 dataset. Our bionic calculation method is applied to the controller, which can realize the continuity of human-computer interaction and the flexibility of manipulator control.
BibTeX
@inproceedings{yang2021dynamic,
  author = {Zhiwen Yang and Du Jiang and Ying Sun and Bo Tao and Xiliang Tong and Guozhang Jiang and Manman Xu and Juntong Yun and Ying Liu and Baojia Chen and Jianyi Kong},
  title = {{Dynamic Gesture Recognition Using Surface EMG Signals Based on Multi-Stream Residual Network}},
  booktitle = {Frontiers in Bioengineering and Biotechnology},
  year = {2021},
  doi = {10.3389/fbioe.2021.779353},
}
Figure