TL;DR
Contrastive Pose-EMG Pre-training (CPEP) bridges the gap between noisy muscle activity (sEMG) and clean hand kinematics by aligning their representations in a shared embedding space. By leveraging a frozen, high-quality pose encoder as an anchor, this framework enables the first-ever zero-shot gesture classification from EMG signals, dramatically improving generalization to unseen gestures and users.
What Problem Does This Paper Address?
Wearable hand gesture recognition using surface electromyography (sEMG) is highly desirable for its low power consumption, privacy preservation, and immunity to camera occlusions. However, sEMG signals are notoriously noisy, user-dependent, and highly variable due to the fine dexterity of hand movements.
While computer vision models thrive on massive video and motion-capture datasets, biosignal models suffer from limited data scale and struggle to generalize to new users or completely unseen gestures. The authors address this bottleneck by asking a fundamental question: Can we leverage high-quality, structured vision/pose data to guide and improve representation learning for weak, noisy biosignals?
Methods
The authors introduce CPEP (Contrastive Pose-EMG Pre-training), a framework inspired by vision-language models like CLIP but tailored for cross-modal biosignal alignment. The pipeline consists of three main stages:
- Modality-Specific Pre-training: They first train independent Masked Autoencoders (MAEs) for both the EMG signals (EMG-MAE) and hand pose data (Pose-MAE) using a standard Transformer backbone. This allows each encoder to learn meaningful unimodal structures from unlabeled data.
- Contrastive Alignment: Next, they freeze the pre-trained pose encoder to act as a high-quality geometric anchor. They append a lightweight projection head to the EMG encoder and train it using a symmetric InfoNCE (contrastive) loss. This pulls the embeddings of paired EMG and pose sequences closer together while pushing mismatched pairs apart.
- Zero-Shot Inference: Because the EMG space is aligned with a structured pose space, gesture classification can be performed without task-specific fine-tuning. For an unseen EMG signal, the model retrieves the top-$k$ nearest pose embeddings in the latent space and uses majority voting to predict the gesture class.
Datasets
The model is trained and evaluated on emg2pose, a large-scale, open-source dataset containing 370 hours of sEMG and synchronized hand pose data across 193 users. The dataset includes 29 different behavioral groups (e.g., making a fist, counting). Hand pose labels were captured using a high-resolution motion capture system, yielding over 80 million pose labels.
To rigorously test generalization, the authors split the dataset into disjoint sets of in-distribution gestures (4 classes) and unseen gestures (4 classes, including challenging movements like "CountingWiggling" and "FingerPinches" that involve severe visual occlusion).
Key Insights and Ideas
- Cross-Modal Supervision as a Kinematic Guide: High-quality modalities (like pose data) capture structural and semantic relationships that are nearly impossible to extract from weak, noisy modalities (like sEMG) alone. Aligning the two forces the EMG encoder to learn rich, pose-informative kinematics.
- The Necessity of a Frozen Anchor: A critical ablation finding showed that the pose encoder must remain frozen during contrastive learning. Allowing it to update alongside the EMG encoder caused the alignment process to fail to converge, proving that a stable geometric anchor is essential when dealing with highly noisy signals.
- Global Context via
[CLS]Tokens: Using an explicit[CLS]token to summarize the sequence outperformed simple average pooling across tokens, demonstrating that global attention mechanisms are better at capturing the complex temporal dynamics of hand gestures.
Key Results
CPEP significantly outperforms existing benchmarks on both in-distribution and out-of-distribution (unseen) gesture classification tasks:
- In-Distribution Performance: Under linear probing, CPEP achieves 78.2% accuracy, outperforming the standard
emg2posebaseline (73.4%) and a purely self-supervised EMG-MAE (34.7%). Crucially, its zero-shot accuracy reaches 75.7% without any task-specific training. - Unseen Gesture Generalization: On completely unseen gestures and users, CPEP achieves 53.6% accuracy with linear probing and 48.1% via zero-shot retrieval. This represents a massive 72% relative improvement over the best-performing baseline models (such as PoseT at 43.3% and emg2pose at 40.5%).
Limitations and Future Work
While CPEP marks a major milestone as the first zero-shot framework for EMG gesture recognition, the authors note a few promising directions for expansion:
- Dataset Diversity: The current evaluation is limited to the
emg2posecorpus. Future work will test CPEP's robustness across other diverse, multi-session EMG datasets. - Multi-Modal Biosignals: The framework could be extended to incorporate other low-power wearable sensors, such as Inertial Measurement Units (IMUs), to create a more holistic representation.
- Broader Applications: Moving beyond isolated hand gestures, the authors aim to explore more complex downstream tasks like continuous human activity recognition and robotic control.
Summary updated