TL;DR
This paper introduces MICA, a novel self-supervised framework for skeleton-based action recognition that tackles modality imbalance and weak feature representations. By jointly modeling intra-modal self-similarity and inter-modal cross-consistency using Feature Modulation and a Frequency-domain Spatial-Temporal block (FreST), MICA achieves state-of-the-art performance on benchmark datasets, significantly improving accuracy for subtle action distinctions.
What Problem Does This Paper Address?
Human Action Recognition (HAR) is crucial for applications like surveillance, autonomous driving, and healthcare. While skeleton-based methods offer a compact representation, they often struggle with subtle action differences (e.g., "reading" vs. "writing") due to lost contextual information. Existing self-supervised and multimodal approaches face challenges like modality imbalance, difficulty integrating spatial-temporal frequency patterns, and weak feature representations. This paper aims to bridge this gap by developing a method that effectively leverages both internal skeleton dynamics and external semantic context, even with limited labeled data.
Methods
MICA is built upon two core components:
- Feature Modulation (FM): This module establishes a shared skeleton-language conceptual domain. It aligns skeleton motion data with textual descriptions by minimizing the information gain between modalities. This helps to create a more robust representation space, improving discrimination between similar actions.
- Frequency-domain Spatial–Temporal Block (FreST): This novel block processes skeleton data in the frequency domain. It adaptively filters sparse yet informative joint movements, emphasizing compact signal energy. FreST uses both global and local frequency filtering to capture structural variations and enhance action representations, effectively suppressing noise and redundancy while preserving discriminative temporal-spatial features.
The framework also incorporates a Modality Harmonizer using cross-attention to further align enhanced frame-level and word-level features. For self-supervised learning, they employ an Skeleton Instance Contrastive Loss based on the InfoNCE objective, using augmented skeleton sequences.
Datasets
The proposed MICA framework was evaluated on three widely used benchmark datasets:
- NTU RGB+D 60 (NTU 60): Contains 56,880 action samples from 40 subjects across 60 action classes, captured with Kinect v2.
- NTU RGB+D 120 (NTU 120): An extension of NTU 60, featuring 114,480 samples from 106 subjects and 120 action classes.
- PKU-MMD: A dataset designed for continuous action detection, including untrimmed video sequences with synchronized RGB, depth, and 25-joint 3D skeleton data. It has two parts with varying detection difficulty.
Key Insights and Ideas
- Frequency Domain is Key for Subtle Actions: By analyzing skeleton data in the frequency domain, MICA can better capture fine-grained temporal dynamics and rhythmic patterns that distinguish similar actions, which are often lost in purely time-domain analysis.
- Language as a Semantic Anchor: Leveraging textual descriptions (even generated ones) as a "semantic anchor" provides crucial context that helps disambiguate actions and improve the overall representation.
- Balancing Modalities is Crucial: The paper highlights that simply fusing modalities isn't enough. MICA's Feature Modulation actively works to minimize information imbalance, leading to better alignment and more discriminative features.
- Adaptive Filtering for Robustness: The adaptive nature of FreST's frequency filtering allows it to dynamically adjust to different action characteristics, making it robust to noise and preserving essential motion signatures.
Key Results
MICA achieved state-of-the-art performance across multiple benchmark datasets and evaluation protocols:
- On the challenging PKU-MMD dataset, MICA showed an improvement of at least 4.6% over methods like CrosSCLR and AimCLR.
- In linear evaluation on NTU-60 X-sub, MICA reached 84.2% accuracy, surpassing AimCLR by 7%.
- In fine-tuning evaluation on NTU-60 X-sub, MICA achieved 86.0%, outperforming 3s-AimCLR by 7%.
- Semi-supervised evaluation also showed significant gains, with MICA achieving 85.0% on NTU-120 X-view with 10% labeled data, outperforming 3s-AimCLR by 7.5%.
- Ablation studies confirmed the effectiveness of both Feature Modulation and the FreST block, with FreST contributing a 3.5% accuracy improvement over the baseline.
Limitations and Future Work
- Sensitivity to Corrupted Skeletons: The performance of MICA might degrade when input skeletons are severely corrupted or noisy, as the feature extraction relies on relatively stable joint trajectories. The authors suggest incorporating denoising or uncertainty-aware modules for future work.
- Computational Overhead: While designed with lightweight attention, the Frequency Modulation and FreST modules introduce some computational and memory overhead, especially for long sequences. Further optimization and model compression techniques are planned for large-scale, real-time deployment.
- Benchmark Limitations: The current evaluation is limited to standard cross-subject and cross-view protocols, making k-fold cross-validation not applicable.
- Scaling to Larger Datasets: The current implementation has been validated on medium-scale benchmarks. Scaling to larger datasets would require further optimization.
Summary updated