Towards Activated Muscle Group Estimation in the Wild

Kunyu Peng, David Schneider, Alina Roitberg, Kailun Yang, Jiaming Zhang, Chen Deng, Kaiyu Zhang, M. Saquib Sarfraz, Rainer Stiefelhagen

TL;DR

This paper introduces Activated Muscle Group Estimation (AMGE) in the wild, a new computer vision task aimed at identifying active muscle regions from standard video without wearable sensors. The authors contribute MuscleMap, a large-scale dataset of 15,000+ HIIT exercise clips, and TransM$^3$E, a multi-modal transformer architecture that achieves state-of-the-art results by intelligently fusing video and skeleton data through novel classification tokens.

What Problem Does This Paper Address?

Understanding human activity is a cornerstone of computer vision, but most research focuses on what someone is doing (e.g., "doing a squat") rather than how their body is internally responding. Knowing which muscles are activated is vital for sports science, physical therapy, and injury prevention.

Traditionally, this requires electromyography (EMG) sensors—wearable devices that are often expensive, cumbersome, and restricted to clinical or lab settings. While some video-based methods exist, they are typically limited to small datasets in controlled environments. The authors tackle the "in-the-wild" challenge: estimating muscle activation from diverse YouTube videos with varying backgrounds, lighting, and camera angles. A key hurdle is generalization: a robust model must be able to estimate muscle activation for new, unseen exercises that weren't part of its training set.

Methods

The authors propose TransM$^3$E, a transformer-based framework designed to bridge the gap between appearance-heavy video data and geometry-focused skeleton data.

Overview of the MuscleMap dataset and the TransM$^3$E model architecture.

Key technical innovations include:

  • Multi-Classification Tokens (MCT): Instead of using a single global classification token, the model employs multiple tokens to better capture the diverse, multi-label nature of muscle activation (where one movement might trigger 10+ different muscle groups).
  • Multi-Classification Tokens Knowledge Distillation (MCTKD): Fusing a video transformer (MViTv2) with a skeleton-based Graph Convolutional Network (GCN) is difficult due to their different structures. MCTKD uses dedicated tokens to transfer knowledge from the skeleton branch (which is better at generalizing to new poses) to the video branch.
  • Multi-Classification Tokens Fusion (MCTF): A specialized mechanism to merge the distilled knowledge and classification tokens for the final prediction.
Figure 3: An overview of the proposed TransM3E architecture.

Datasets

The paper introduces MuscleMap135, the first large-scale "in-the-wild" AMGE dataset:

  • Scale: 15,004 video clips.
  • Diversity: 135 different High-Intensity Interval Training (HIIT) activities.
  • Labels: 20 distinct muscle groups (e.g., biceps, quadriceps, obliques) annotated with binary activation status by sports science experts.
  • Modalities: Provides RGB video, RGB difference, optical flow, and 2D skeleton data.
Figure 2: An overview of the number of samples and the number of activity types per muscle region (@R), depicted at the top left and the top right.

Key Insights and Ideas

  • Complementary Modalities: The authors found that video-based models excel at recognizing known activities but struggle with new ones because they overfit to specific backgrounds. Conversely, skeleton-based models generalize better to new movements because they ignore the background, but they lack the fine-grained visual cues (like muscle bulge or skin tension) that RGB video provides.
  • The Generalization Gap: Existing action recognition models perform poorly when asked to estimate muscle activation for exercises they haven't seen before. TransM$^3$E specifically targets this by using skeleton data to "anchor" the model's understanding of body mechanics.
  • Token-Level Distillation: By performing knowledge distillation on specific tokens rather than full feature maps, the model avoids the "alignment issue" typically found when trying to force two very different neural architectures to talk to each other.

Key Results

TransM$^3$E sets a new benchmark on the MuscleMap dataset, achieving a mean Average Precision (mAP) of 81.0%.

  • Superior Generalization: It outperforms the strong MViTv2-S baseline by 2.8% mAP on unseen activity types, proving that the multi-modal fusion strategy effectively helps the model handle new exercises.
  • Qualitative Focus: Grad-CAM visualizations show that TransM$^3$E focuses more accurately on the relevant body regions compared to standard video transformers, which often get distracted by background elements.
Figure 5: Qualitative results for the MViTv2-S [38] and TransM3E. GradCam [64] visualization is given.

Limitations and Future Work

  • Binary Labels: Currently, the dataset only indicates if a muscle is "on" or "off." It does not capture the intensity of the activation, which is crucial for advanced athletic training.
  • Temporal Resolution: Annotations are provided per video clip rather than per frame. Future work could involve frame-level estimation to track muscle activation changes throughout a single repetition.
  • Missing Modalities: The authors suggest exploring "missing-modality" scenarios, where the model might only have access to video or only to skeletons at test time.

Summary updated

Abstract

In this paper, we tackle the new task of video-based Activated Muscle Group Estimation (AMGE) aiming at identifying active muscle regions during physical activity in the wild. To this intent, we provide the MuscleMap dataset featuring >15K video clips with 135 different activities and 20 labeled muscle groups. This dataset opens the vistas to multiple video-based applications in sports and rehabilitation medicine under flexible environment constraints. The proposed MuscleMap dataset is constructed with YouTube videos, specifically targeting High-Intensity Interval Training (HIIT) physical exercise in the wild. To make the AMGE model applicable in real-life situations, it is crucial to ensure that the model can generalize well to numerous types of physical activities not present during training and involving new combinations of activated muscles. To achieve this, our benchmark also covers an evaluation setting where the model is exposed to activity types excluded from the training set. Our experiments reveal that the generalizability of existing architectures adapted for the AMGE task remains a challenge. Therefore, we also propose a new approach, TransM3E, which employs a multi-modality feature fusion mechanism between both the video transformer model and the skeleton-based graph convolution model with novel cross-modal knowledge distillation executed on multi-classification tokens. The proposed method surpasses all popular video classification models when dealing with both, previously seen and new types of physical activities. The database and code can be found at https://github.com/KPeng9510/MuscleMap.
BibTeX
@inproceedings{peng2024towards,
  author = {Kunyu Peng and David Schneider and Alina Roitberg and Kailun Yang and Jiaming Zhang and Chen Deng and Kaiyu Zhang and M. Saquib Sarfraz and Rainer Stiefelhagen},
  title = {{Towards Activated Muscle Group Estimation in the Wild}},
  booktitle = {ACM Multimedia},
  year = {2024},
  doi = {10.1145/3664647.3680630},
}
Figure