TL;DR
This paper introduces Activated Muscle Group Estimation (AMGE) in the wild, a new computer vision task aimed at identifying active muscle regions from standard video without wearable sensors. The authors contribute MuscleMap, a large-scale dataset of 15,000+ HIIT exercise clips, and TransM$^3$E, a multi-modal transformer architecture that achieves state-of-the-art results by intelligently fusing video and skeleton data through novel classification tokens.
What Problem Does This Paper Address?
Understanding human activity is a cornerstone of computer vision, but most research focuses on what someone is doing (e.g., "doing a squat") rather than how their body is internally responding. Knowing which muscles are activated is vital for sports science, physical therapy, and injury prevention.
Traditionally, this requires electromyography (EMG) sensors—wearable devices that are often expensive, cumbersome, and restricted to clinical or lab settings. While some video-based methods exist, they are typically limited to small datasets in controlled environments. The authors tackle the "in-the-wild" challenge: estimating muscle activation from diverse YouTube videos with varying backgrounds, lighting, and camera angles. A key hurdle is generalization: a robust model must be able to estimate muscle activation for new, unseen exercises that weren't part of its training set.
Methods
The authors propose TransM$^3$E, a transformer-based framework designed to bridge the gap between appearance-heavy video data and geometry-focused skeleton data.
Key technical innovations include:
- Multi-Classification Tokens (MCT): Instead of using a single global classification token, the model employs multiple tokens to better capture the diverse, multi-label nature of muscle activation (where one movement might trigger 10+ different muscle groups).
- Multi-Classification Tokens Knowledge Distillation (MCTKD): Fusing a video transformer (MViTv2) with a skeleton-based Graph Convolutional Network (GCN) is difficult due to their different structures. MCTKD uses dedicated tokens to transfer knowledge from the skeleton branch (which is better at generalizing to new poses) to the video branch.
- Multi-Classification Tokens Fusion (MCTF): A specialized mechanism to merge the distilled knowledge and classification tokens for the final prediction.
Datasets
The paper introduces MuscleMap135, the first large-scale "in-the-wild" AMGE dataset:
- Scale: 15,004 video clips.
- Diversity: 135 different High-Intensity Interval Training (HIIT) activities.
- Labels: 20 distinct muscle groups (e.g., biceps, quadriceps, obliques) annotated with binary activation status by sports science experts.
- Modalities: Provides RGB video, RGB difference, optical flow, and 2D skeleton data.
Key Insights and Ideas
- Complementary Modalities: The authors found that video-based models excel at recognizing known activities but struggle with new ones because they overfit to specific backgrounds. Conversely, skeleton-based models generalize better to new movements because they ignore the background, but they lack the fine-grained visual cues (like muscle bulge or skin tension) that RGB video provides.
- The Generalization Gap: Existing action recognition models perform poorly when asked to estimate muscle activation for exercises they haven't seen before. TransM$^3$E specifically targets this by using skeleton data to "anchor" the model's understanding of body mechanics.
- Token-Level Distillation: By performing knowledge distillation on specific tokens rather than full feature maps, the model avoids the "alignment issue" typically found when trying to force two very different neural architectures to talk to each other.
Key Results
TransM$^3$E sets a new benchmark on the MuscleMap dataset, achieving a mean Average Precision (mAP) of 81.0%.
- Superior Generalization: It outperforms the strong MViTv2-S baseline by 2.8% mAP on unseen activity types, proving that the multi-modal fusion strategy effectively helps the model handle new exercises.
- Qualitative Focus: Grad-CAM visualizations show that TransM$^3$E focuses more accurately on the relevant body regions compared to standard video transformers, which often get distracted by background elements.
Limitations and Future Work
- Binary Labels: Currently, the dataset only indicates if a muscle is "on" or "off." It does not capture the intensity of the activation, which is crucial for advanced athletic training.
- Temporal Resolution: Annotations are provided per video clip rather than per frame. Future work could involve frame-level estimation to track muscle activation changes throughout a single repetition.
- Missing Modalities: The authors suggest exploring "missing-modality" scenarios, where the model might only have access to video or only to skeletons at test time.
Summary updated