HYDRA-HGR: A Hybrid Transformer-Based Architecture for Fusion of Macroscopic and Microscopic Neural Drive Information

Mansooreh Montazerin, Elahe Rahimian, Farnoosh Naderkhani, Seyed Farokh Atashzar, Hamid Alinejad-Rokny, Arash Mohammadi

TL;DR

HYDRA-HGR is a novel deep learning framework that significantly improves hand gesture recognition by fusing "macroscopic" raw muscle signals with "microscopic" neural firing patterns. Using a dual-path Vision Transformer (ViT) architecture, the system achieves a high accuracy of 94.86%, outperforming models that rely on only one type of data.

What Problem Does This Paper Address?

Developing reliable Human-Machine Interfaces (HMIs) for prosthetics and "Cyber-Physical-Human" systems depends on accurately recognizing hand gestures from surface Electromyogram (sEMG) signals. Traditionally, researchers have focused on macroscopic information—the raw electrical activity recorded from the skin.

While recent advancements in High-Density sEMG (HD-sEMG) allow for microscopic analysis (decomposing signals to see individual motor unit activities), this "neural drive" information is rarely used for gesture recognition due to its computational complexity. The authors argue that ignoring either level of information limits the system's accuracy and robustness. HYDRA-HGR aims to bridge this gap by creating a hybrid model that listens to both the "broad strokes" of muscle activity and the "fine details" of neural instructions.

Methods

The researchers proposed a hybrid architecture consisting of two parallel Vision Transformer (ViT) paths:

  1. The Macro Path: This path processes pre-processed raw HD-sEMG signals. The data is treated as a 3D image (time × 2D spatial grid of electrodes). It uses a ViT to extract broad spatial and temporal features.
  2. The Micro Path: This path uses Blind Source Separation (BSS) to decompose the HD-sEMG signals into individual Motor Unit Action Potentials (MUAPs). These are converted into 2D "peak-to-peak" images that represent the relative activation levels of specific neural sources. A second ViT processes these microscopic images.

The Fusion: The "class tokens" (the learned summary vectors) from both the Macro and Micro ViTs are concatenated and fed into a final Fully Connected (FC) layer for classification. This allows the model to make a final decision based on the synergy between both data types.

Datasets

The study utilized a recently released, large-scale HD-sEMG dataset:

  • Participants: 20 non-disabled individuals.
  • Gestures: 65 isometric hand movements (66 total gestures including repetitions).
  • Hardware: Two 8×8 electrode grids (128 channels total) sampled at 2,048 Hz.
  • Structure: 5 repetitions per gesture, evaluated using 5-fold cross-validation.

Key Insights and Ideas

  • Complementary Information: The core "aha moment" is that macroscopic and microscopic data are not redundant. The Macro path captures the overall muscle geometry and intensity, while the Micro path captures the specific neural "signatures" of a gesture.
  • The Power of Transformers: While CNNs and RNNs are common in this field, the authors leverage Vision Transformers (ViTs). ViTs use an attention mechanism that is particularly adept at finding long-range correlations in the spatial-temporal data of muscle grids.
  • Minimalist Micro-Data: Interestingly, the Micro path only requires a small number of underlying sources (a maximum of 7 in this study) to provide a significant performance boost, suggesting that full neural decomposition might not be necessary for high-accuracy HGR.

Key Results

  • Superior Accuracy: The HYDRA-HGR framework achieved an average accuracy of 94.86% for a 250 ms window.
  • Performance Gain: This represents a 5.52% improvement over the standalone Macro path and an 8.22% improvement over the standalone Micro path.
  • Consistency: As shown in the researchers' boxplots, the hybrid model exhibited lower variance and higher overall stability across different subjects compared to single-path models.

Limitations and Future Work

  • Computational Overhead: The decomposition stage (BSS) required to extract microscopic information adds computational weight. The authors acknowledge that high latency and training times remain challenges for real-time HMI applications.
  • Subject Diversity: The dataset consisted of non-disabled participants. Future work needs to validate this architecture on data from amputees or individuals with neuromuscular disorders, where signal patterns may be significantly different.
  • Real-time Implementation: While the 250 ms window is promising, the end-to-end latency of the decomposition plus the dual-ViT inference needs further optimization for seamless prosthetic control.

Summary updated

Abstract

Development of advance surface Electromyogram (sEMG)-based Human-Machine Interface (HMI) systems is of paramount importance to pave the way towards emergence of futuristic Cyber-Physical-Human (CPH) worlds. In this context, the main focus of recent literature was on development of different Deep Neural Network (DNN)-based architectures that perform Hand Gesture Recognition (HGR) at a macroscopic level (i.e., directly from sEMG signals). At the same time, advancements in acquisition of High-Density sEMG signals (HD-sEMG) have resulted in a surge of significant interest on sEMG decomposition techniques to extract microscopic neural drive information. However, due to complexities of sEMG decomposition and added computational overhead, HGR at microscopic level is less explored than its aforementioned macroscopic-level, DNN-based counterparts. In this regard, we propose the HYDRA-HGR framework, which is a hybrid model for HGR that simultaneously extracts a set of temporal and spatial features through its two independent Vision Transformer (ViT)-based parallel architectures (the so called Macro and Micro paths). The Macro Path is trained directly on the pre-processed HD-sEMG signals, while the Micro path is fed with the p-to-p values of the extracted Motor Unit Action Potentials (MUAPs) of each source. Extracted features at macroscopic and microscopic levels are then coupled via a Fully Connected (FC) fusion layer for final gesture classification. We evaluate the proposed hybrid HYDRA-HGR framework through a recently released HD-sEMG dataset, and show that it significantly outperforms its stand-alone counterparts. The proposed HYDRA-HGR framework achieves average accuracy of 94.86% for the 250 ms window size, which is 5.52 % and 8.22 % higher than that of the Macro and Micro paths, respectively.
BibTeX
@inproceedings{montazerin2023hydra,
  author = {Mansooreh Montazerin and Elahe Rahimian and Farnoosh Naderkhani and Seyed Farokh Atashzar and Hamid Alinejad-Rokny and Arash Mohammadi},
  title = {{HYDRA-HGR: A Hybrid Transformer-Based Architecture for Fusion of Macroscopic and Microscopic Neural Drive Information}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10096192},
}
Figure