Zero-shot Skeleton-based Action Recognition via Mutual Information Estimation and Maximization

Yujie Zhou, Wenwen Qiang, Anyi Rao, Ning Lin, Bing Su, Jiaqi Wang

TL;DR

SMIE (Skeleton-based Mutual Information Estimation) is a new framework for zero-shot action recognition that moves beyond simple feature mapping. By maximizing the Mutual Information (MI) between visual skeleton sequences and semantic text labels, and enforcing a temporal constraint that rewards the model for identifying keyframes, it achieves state-of-the-art performance on major benchmarks like NTU-120.

What Problem Does This Paper Address?

Zero-shot action recognition (ZAR) is the "holy grail" of activity understanding: recognizing actions (like "skipping") even if the model was only trained on others (like "walking" or "jumping"). This is usually done by bridging the gap between visual features (the movement) and semantic features (the word "skipping").

The authors identify two major flaws in current skeleton-based ZAR methods:

  1. Distribution Ignorance: Most models try to map a visual vector directly to a text vector (point-to-point). This ignores the global structure of how different actions relate to each other in their respective spaces.
  2. Temporal Negligence: Skeleton data is a time series, but many models pool the entire sequence into a single vector, losing the "story" of the movement. They fail to capture that some frames (keyframes) are more semantically important than others.

Methods

The proposed SMIE framework introduces two novel modules to solve these issues:

  • Global Alignment Module: Instead of a simple projection, the authors use a neural network to estimate and maximize the Mutual Information (MI) between the visual and semantic distributions. Using a Jensen-Shannon Divergence (JSD) estimator, the model learns to maximize the similarity score for paired visual-semantic data while minimizing it for unpaired (negative) samples. This aligns the "shapes" of the two data distributions, making it easier to generalize to unseen classes.
  • Temporal Constraint Module: To capture dynamics, the model uses "bidirectional motion attention" to identify keyframes (frames with the most movement). It then creates an "attention-masked" version of the sequence where these keyframes are hidden.
  • Hinge Loss: The authors apply a specific loss function that forces the MI of the full sequence to be higher than the MI of the masked sequence by a set margin. This teaches the model that specific temporal parts of the action are critical for its semantic identity.

Datasets

The authors evaluated their method on three large-scale skeleton action datasets:

  • NTU-RGB+D 60: 56k sequences across 60 categories.
  • NTU-RGB+D 120: An extension of the above with 114k sequences and 120 categories.
  • PKU-MMD: Nearly 20k samples in 51 categories.

They used standard "seen/unseen" splits (e.g., training on 55 classes and testing on 5) to prove the zero-shot capability.

Key Insights and Ideas

  • MI over Projections: The shift from "mapping points" to "aligning distributions" via Mutual Information is the core "aha" moment. It treats the relationship between vision and language as a statistical correlation rather than a rigid geometric translation.
  • The "Keyframe" Hypothesis: By penalizing the model when it loses information from high-motion frames, the authors successfully bake "common sense" into the architecture—the idea that the peak of a jump is more important than the standing start.
  • ChatGPT as a Feature Booster: In a fascinating experiment, the authors used ChatGPT to expand simple labels (e.g., "Wear jacket") into detailed descriptions ("the act of putting on a garment..."). This richer semantic input led to a massive performance jump (nearly 7% on NTU-60), suggesting that the bottleneck in ZAR is often the brevity of the text labels themselves.

Key Results

SMIE significantly outperformed existing state-of-the-art (SOTA) methods like SynSE and CADA-VAE:

  • NTU-60: Achieved 40.18% accuracy on the 48/12 split, a ~7% absolute improvement over the previous SOTA (SynSE at 33.30%).
  • NTU-120: Achieved 45.30% on the 96/24 split (vs. 38.70% for SynSE).
  • Ablation Studies: Removing the Temporal Constraint Module caused a drop of about 1-1.4% across datasets, proving that temporal modeling provides a consistent edge.

Limitations and Future Work

The authors noted two main challenges:

  1. Short Actions: Very brief actions like "yawning" or "jumping up" are still frequently misclassified because the temporal constraint has less "room" to work with in short sequences.
  2. Fine-grained Movements: Actions involving subtle hand movements (like "tearing up paper") are difficult to distinguish using only skeleton joints, as the sensors often lack the precision to capture finger-level detail.

Future work will likely focus on better handling these subtle, fine-grained interactions and further exploring how large language models (LLMs) can provide even better semantic anchors for visual data.

Summary updated

Abstract

Zero-shot skeleton-based action recognition aims to recognize actions of unseen categories after training on data of seen categories. The key is to build the connection between visual and semantic space from seen to unseen classes. Previous studies have primarily focused on encoding sequences into a singular feature vector, with subsequent mapping the features to an identical anchor point within the embedded space. Their performance is hindered by 1) the ignorance of the global visual/semantic distribution alignment, which results in a limitation to capture the true interdependence between the two spaces. 2) the negligence of temporal information since the frame-wise features with rich action clues are directly pooled into a single feature vector. We propose a new zero-shot skeleton-based action recognition method via mutual information (MI) estimation and maximization. Specifically, 1) we maximize the MI between visual and semantic space for distribution alignment; 2) we leverage the temporal information for estimating the MI by encouraging MI to increase as more frames are observed. Extensive experiments on three large-scale skeleton action datasets confirm the effectiveness of our method.
BibTeX
@inproceedings{zhou2023zero,
  author = {Yujie Zhou and Wenwen Qiang and Anyi Rao and Ning Lin and Bing Su and Jiaqi Wang},
  title = {{Zero-shot Skeleton-based Action Recognition via Mutual Information Estimation and Maximization}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3611888},
}

Loading related content...

Figure