Abstract
Hand gesture recognition (HGR) plays a pivotal role in natural and intuitive human-computer interactions. Recent HGR methods focus on recognizing gestures from vision-based images or videos. However, vision-based methods are limited in recognizing micro hand gestures (MHGs) (e.g., pinch within 1cm) and gestures with occluded fingers. To address these issues, combined with the electromyography (EMG) technique, we propose BeyondVision, an EMG-driven MHG recognition system based on deep learning. BeyondVision consists of a wristband-style EMG sampling device and a tailored lightweight neural network BV-Net that can accurately translate EMG signals of MHGs to control commands in real-time. Moreover, we propose a post-processing mechanism and a weight segmentation algorithm to effectively improve the accuracy rate of MHG recognition. Subjective and objective experimental results show that our approach achieves over 95% average recognition rate, 2000Hz sampling frequency, and real-time micro gesture recognition.
Introduction
Most modern hand gesture recognition (HGR) systems rely on computer vision technologies. While effective for large-scale movements, these systems face significant challenges with occlusion and scale. Specifically, "micro hand gestures" (MHGs)—subtle movements like a 1cm finger pinch—are difficult for cameras to capture reliably, especially if fingers are obscured. Furthermore, MHGs are highly desirable for long-term human-computer interaction (HCI) as they minimize user fatigue compared to broad arm movements.
Electromyography (EMG) offers a solution by measuring the electrical activity of muscles directly. This approach is immune to visual obstructions and sensitive enough to detect minute muscle contractions. BeyondVision bridges the gap in current HGR systems by capturing the "muscle intent" that precedes or accompanies even the slightest physical movement.
System Design and Methodology
BeyondVision employs a hardware-software co-design to achieve high-fidelity gesture recognition.
Hardware and Data Acquisition
The system utilizes a custom-built wearable wristband equipped with 16-channel EMG sensors. This high-density array captures the complex interplay of 14 muscles and 2 nerves in the forearm at a high sampling frequency of 2000Hz. This frequency is critical for capturing the high-frequency components of micro-movements that define MHGs.
Signal Processing and BV-Net
Raw signals undergo pre-processing via a 10Hz high-pass filter and a Root Mean Square (RMS) algorithm to enhance signal patterns while suppressing noise. The core of the system is BV-Net, a lightweight Convolutional Neural Network (CNN). BV-Net incorporates "EMG Blocks" inspired by ShuffleNet for computational efficiency and "CBM" modules (Convolution, Batch Normalization, and Mish activation). To address the difficulty of distinguishing similar micro-gestures, a modified Focal Loss function with L2 regularization is used during training.
Weight Segmentation (WS) Algorithm
A key innovation is the Weight Segmentation algorithm, which introduces dynamic temporal context. Instead of classifying static snapshots, the WS algorithm uses an attention window to segment the EMG spectrum and evaluates sequences of predictions, significantly improving stability and accuracy. This algorithm alone provided a 2.67% boost in average accuracy.
Experimental Evaluation
The researchers developed a comprehensive MHG dataset involving 20 participants and 14 distinct gesture classes, including specific finger pinches, slides, and pushes. Data augmentation techniques like "White Noise Integration" and random amplitude scaling were used to simulate natural muscle variability.
Key Results
- High Accuracy: BeyondVision achieved an average recognition rate of 95.66% across 14 gesture classes.
- Superiority over Vision-Based Systems: When compared to the state-of-the-art Vision Transformer for HGR (ViT-HGR), which scored 76.4%, BeyondVision showed a massive improvement, particularly in "slide" gestures where vision systems often struggle.
- Real-time Performance: The system was successfully demonstrated in real-time applications, such as controlling the game Subway Surfer using subtle finger movements with minimal latency.
Conclusion and Future Work
BeyondVision demonstrates that EMG-driven systems can effectively capture the subtle language of micro-gestures where vision-based systems fail. While the current system is "dynamic-oriented" (requiring active muscle change), future work will explore hybrid models combining vision for global positioning and EMG for precise, occluded interactions. Efforts will also focus on miniaturizing the hardware and improving "zero-shot" learning for new users without calibration.
Summary updated