USDRL: Unified Skeleton-Based Dense Representation Learning with Multi-Grained Feature Decorrelation

Wanjiang Weng, Hongsong Wang, Junbo Wang, Lei He, Guo-Sen XieAAAI 2025 · DOI

Abstract

Self-supervised representation learning for skeleton-based action recognition has traditionally relied on contrastive learning with negative samples or masked modeling. However, these approaches often require complex memory banks or focus solely on global representations, limiting their effectiveness for dense prediction tasks like action detection. We present USDRL, a framework that shifts the paradigm toward multi-grained feature decorrelation. By eliminating the need for negative samples and maintaining dense spatio-temporal features, USDRL achieves state-of-the-art performance in both action recognition and fine-grained action detection.

Introduction

Skeleton-based action recognition is a critical task in computer vision, with applications ranging from human-computer interaction to surveillance. Recent advances in self-supervised learning have primarily utilized two strategies: Masked Sequence Modeling (MSM) and Contrastive Learning (CL). While effective, CL typically requires a large number of negative samples to avoid representation collapse, necessitating resource-intensive components like memory banks and momentum encoders. Furthermore, most existing methods compress entire video sequences into a single global vector, which is insufficient for "dense" tasks such as temporal action detection where frame-level precision is required.

Comparison of self-supervised learning paradigms for skeleton data: Contrastive Learning, Masked Sequence Modeling, and the proposed Feature Decorrelation.

USDRL (Unified Skeleton-based Dense Representation Learning) addresses these challenges by rethinking the learning objective. Instead of comparing samples against a set of negatives, USDRL focuses on decorrelating features across temporal, spatial, and instance domains to learn non-redundant, informative representations.

Methodology

The USDRL architecture consists of two primary components: a Dense Spatio-Temporal Encoder (DSTE) and a Multi-Grained Feature Decorrelation module.

Dense Spatio-Temporal Encoder (DSTE)

To preserve fine-grained information, the DSTE avoids aggressive pooling. It processes skeleton sequences through two parallel attention mechanisms:

  • Dense Shift Attention (DSA): This module captures long-range temporal dependencies by applying a "DenseShift" operation, allowing the model to relate distant frames without the computational overhead of full global attention.
  • Convolutional Attention (CA): This focuses on local spatio-temporal relationships, ensuring that the subtle movements of specific joints are accurately represented.

Multi-Grained Feature Decorrelation

The core innovation of USDRL is its approach to feature learning. Rather than using negative samples, it employs a loss function designed to maximize information content within the feature vectors themselves.

The USDRL framework architecture, highlighting the Dense Spatio-Temporal Encoder and the Multi-Grained Feature Decorrelation process across temporal, spatial, and instance projectors.

The model applies decorrelation across three distinct domains:

  1. Temporal Domain: Ensures that features remain consistent and informative over the duration of the action.
  2. Spatial Domain: Focuses on the relationships between different skeletal joints.
  3. Instance Domain: Ensures that the overall representation of an action sequence is unique and discriminative.

The total loss function combines Invariance, which ensures that different augmentations of the same skeleton sequence result in similar features, and Redundancy Reduction, which forces different dimensions of the feature vector to capture independent information. This approach effectively prevents representation collapse without the need for a memory bank.

Experimental Results

USDRL was evaluated on several benchmark datasets, including NTU-RGB+D 60 & 120 for action recognition and PKU-MMD I & II for action detection.

Action Recognition

On the NTU-60 (cross-subject) benchmark, USDRL achieved an accuracy of 85.2% using only joint data. This performance significantly exceeds previous feature-decorrelation methods and rivals complex contrastive frameworks, despite USDRL's more streamlined architecture.

Dense Action Detection

The "dense" nature of USDRL's representations provides a significant advantage in action detection tasks. On the PKU-MMD I dataset, the model achieved a mean Average Precision (mAP) of 75.7%, a substantial improvement over the previous state-of-the-art of 61.8%. This result underscores the importance of maintaining spatio-temporal resolution throughout the encoding process.

Conclusion

USDRL demonstrates that high-quality skeleton representations can be learned without the complexity of negative-based contrastive learning. By leveraging multi-grained feature decorrelation and a dense encoder, the framework provides a robust and efficient solution for both global classification and fine-grained detection tasks. Future work may explore extending this decorrelation logic to multi-modal data streams and more complex, multi-subject environments.

Summary updated

BibTeX
@inproceedings{weng2025usdrl,
  author = {Wanjiang Weng and Hongsong Wang and Junbo Wang and Lei He and Guo-Sen Xie},
  title = {{USDRL: Unified Skeleton-Based Dense Representation Learning with Multi-Grained Feature Decorrelation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i8.32899},
}
Figure