Study on Gesture Recognition Method with Two-Stream Residual Network Fusing sEMG Signals and Acceleration Signals

Zhigang Hu, Shen Wang, Cuisi Ou, Aoru Ge, Xiangpan LiSensors 2024 · DOI

TL;DR

This paper introduces a novel two-stream residual neural network that fuses surface electromyography (sEMG) and hand acceleration signals for improved gesture recognition. By processing these two distinct data streams separately and then combining their learned features, the model achieves a high accuracy of 88.25% for 49 gestures, outperforming single-stream approaches and offering a more robust solution for human-computer interaction.

What Problem Does This Paper Address?

Recognizing human gestures using bioelectrical signals like sEMG is a promising area for human-computer interaction, particularly in fields like rehabilitation and prosthetics. However, traditional machine learning methods often struggle with the complex, non-linear nature of these signals, requiring extensive manual feature engineering which can be challenging and may not yield optimal results. While deep learning has shown promise, effectively leveraging multiple, complementary data sources like sEMG and motion data (acceleration) to capture the full picture of a gesture remains an area for improvement. This paper aims to bridge this gap by developing a deep learning architecture that can effectively fuse these multi-modal signals for more accurate and reliable gesture recognition.

Methods

The core of this work is a two-stream residual network architecture. This network comprises two parallel branches: one dedicated to processing surface electromyography (sEMG) signals and the other to processing hand acceleration signals. Both streams utilize residual blocks, a key innovation in deep learning that helps mitigate the vanishing gradient problem in deep networks, allowing for more effective learning of complex features.

Before entering the network, the raw sEMG and acceleration data undergo a series of preprocessing steps:

  • Filtering and denoising: To remove unwanted noise and interference.
  • Z-score normalization: To standardize the amplitude of signals across different individuals and sessions.
  • Signal segmentation: Using a sliding window approach to extract meaningful temporal segments of the signals.

A crucial addition to the network is an attention mechanism, specifically the Efficient Channel Attention (ECA) module. This module is applied after global average pooling in each stream to help the network focus on the most relevant features within each data modality, suppressing less important ones. Finally, the deep features extracted from both the sEMG and acceleration streams are fused to generate a comprehensive representation for gesture classification.

Datasets

The study utilizes the NinaPro DB2 public dataset. This dataset is a multimodal collection containing sEMG, inertia, kinematic, and force data from 40 subjects performing 49 different hand gestures and rest states. The data was acquired using 12 Trigno wireless electrodes, each equipped with a triaxial accelerometer, at a sampling rate of 2000 Hz. The dataset was split into training and testing sets, with 2/3 of repetitions used for training and 1/3 for testing.

Key Insights and Ideas

  • Multi-modal fusion is key: Combining sEMG (physiological muscle activity) with acceleration (kinematic motion) provides a richer, more comprehensive understanding of a gesture than either signal alone. This synergy is crucial for improving recognition accuracy.
  • Residual networks for depth and stability: The use of residual blocks allows for deeper network architectures, enabling the extraction of more intricate features without suffering from gradient vanishing issues, leading to more stable training and better performance.
  • Attention for feature refinement: The ECA attention mechanism effectively guides the network to focus on the most informative channels within each data stream, enhancing the discriminative power of the learned features.
  • Optimal parameter tuning is critical: The study highlights the significant impact of parameters like window length and dropout rate on performance, demonstrating that careful selection (e.g., 200ms window, 0.5 dropout) is vital for maximizing accuracy.

Key Results

  • The proposed two-stream residual network with ECA attention achieved an average recognition accuracy of 88.25% for 49 gestures on the NinaPro DB2 dataset.
  • This multi-modal approach significantly outperformed single-stream models:
  • EMG-only input: 82.75% accuracy.
  • Acceleration-only input: 80.5% accuracy.
  • The two-stream model showed an improvement of approximately 6% over EMG-only and 8% over acceleration-only.
  • The ECA attention mechanism demonstrated superior performance compared to a baseline without attention and the SE-Net module, indicating its effectiveness in feature modeling.
  • The model exhibited stable performance, with average test set accuracy stabilizing around 88% and a gradually converging loss function during training.

Limitations and Future Work

  • Inter-subject variability: While the study analyzed data from 40 subjects and computed average performance to mitigate individual differences, further exploration into inter-subject variability and its impact on model generalization is suggested.
  • Real-time performance: Although the study aimed for real-time capabilities and found optimal parameters that suggest low delay, further optimization for extremely low-latency applications could be explored.
  • Gesture complexity: The current study focused on 49 gestures. Future work could investigate the model's performance on a larger and more diverse set of gestures, including more subtle or complex movements.
  • Exploring other fusion strategies: While the current fusion method is effective, investigating alternative fusion techniques could potentially yield further improvements.

Summary updated

Abstract

Currently, surface EMG signals have a wide range of applications in human-computer interaction systems. However, selecting features for gesture recognition models based on traditional machine learning can be challenging and may not yield satisfactory results. Considering the strong nonlinear generalization ability of neural networks, this paper proposes a two-stream residual network model with an attention mechanism for gesture recognition. One branch processes surface EMG signals, while the other processes hand acceleration signals. Segmented networks are utilized to fully extract the physiological and kinematic features of the hand. To enhance the model's capacity to learn crucial information, we introduce an attention mechanism after global average pooling. This mechanism strengthens relevant features and weakens irrelevant ones. Finally, the deep features obtained from the two branches of learning are fused to further improve the accuracy of multi-gesture recognition. The experiments conducted on the NinaPro DB2 public dataset resulted in a recognition accuracy of 88.25% for 49 gestures. This demonstrates that our network model can effectively capture gesture features, enhancing accuracy and robustness across various gestures. This approach to multi-source information fusion is expected to provide more accurate and real-time commands for exoskeleton robots and myoelectric prosthetic control systems, thereby enhancing the user experience and the naturalness of robot operation.
BibTeX
@inproceedings{hu2024study,
  author = {Zhigang Hu and Shen Wang and Cuisi Ou and Aoru Ge and Xiangpan Li},
  title = {{Study on Gesture Recognition Method with Two-Stream Residual Network Fusing sEMG Signals and Acceleration Signals}},
  booktitle = {Sensors},
  year = {2024},
  doi = {10.3390/s24092702},
}
Figure