Skeleton-based Action Recognition with Non-linear Dependency Modeling and Hilbert-Schmidt Independence Criterion

Haipeng Chen, Yuheng Yang, Yingda LyuAAAI 2025 · DOI

Introduction

Human action recognition using skeletal data has gained significant traction due to its robustness against environmental factors like lighting and background clutter. Most state-of-the-art methods utilize Graph Convolutional Networks (GCNs) to model the spatial-temporal dynamics of joint coordinates. However, these methods typically rely on the physical "tree-like" structure of the human skeleton, which imposes two significant limitations. First, fixed physical topologies struggle to capture long-range dependencies between non-adjacent joints (e.g., the coordination between hands and feet). Second, traditional classifiers often fail to accurately estimate probability densities in high-dimensional feature spaces—a phenomenon known as the "curse of dimensionality."

This paper proposes a novel framework that addresses these bottlenecks through non-linear dependency refinement and a classification objective based on the Hilbert-Schmidt Independence Criterion (HSIC).

Methodology

Non-linear Dependency Refinement

To move beyond the constraints of physical connectivity, the proposed model calculates dependencies between all joint pairs using a Gaussian correlation function. By adjusting the kernel width ($\delta$), the model can dynamically refine the skeletal graph. A larger $\delta$ allows the network to capture global coordination across distant joints, while a smaller $\delta$ focuses on local joint interactions. This process transforms the static physical graph into a "Novel Graph" that better reflects the functional dependencies required for specific actions.

HSIC-based Classification Framework

To mitigate the challenges of high-dimensional feature classification, the authors introduce an HSIC-based objective. Instead of performing classification in raw Euclidean space, motion features are mapped into a Reproducing Kernel Hilbert Space (RKHS) using a Matern covariance function. HSIC is then used to measure the statistical dependence between these mapped features and the action labels. Because HSIC is dimension-agnostic, it maintains discriminative power even as the dimensionality of the input data increases, leading to more robust classification.

Overview of the proposed framework, including the dependency refinement module and the HSIC-based classification objective.

Multi-Stream Learning and Distillation

The architecture employs a dual-stream approach consisting of a "Base Model" and an "Auxiliary Model." The auxiliary model captures additional motion cues, which are distilled into the base model using Kullback-Leibler (KL) divergence. The final prediction is an ensemble of models trained with varying kernel widths, ensuring a comprehensive capture of both fine-grained and global motion patterns.

Experimental Results

The framework was evaluated on three major benchmarks: NTU RGB+D 60, NTU RGB+D 120, and Northwestern-UCLA. The model achieved state-of-the-art performance across all datasets. Notably, on the NTU RGB+D 120 dataset, it reached 90.6% accuracy in the Cross-Subject setting and 91.7% in the Cross-Setup setting.

Ablation studies confirmed the necessity of both the dependency refinement and the HSIC components. Removing the HSIC module resulted in a performance drop of approximately 1.4%, highlighting its role in effective feature separation.

t-SNE visualization comparing feature distributions without HSIC versus the proposed framework across five action classes.

As shown in the t-SNE visualizations, the proposed framework produces much tighter and more distinct clusters for different action classes compared to a baseline without the HSIC component. This visual evidence supports the claim that mapping features to the Hilbert space enhances the model's ability to distinguish between complex human actions.

Conclusion

By integrating non-linear dependency modeling with an HSIC-based classification framework, this research successfully overcomes the structural and dimensional limitations of traditional skeleton-based GCNs. The resulting model sets new performance benchmarks and demonstrates the power of kernel-based methods in high-dimensional motion analysis. Future work will focus on extending this framework to unsupervised and few-shot learning scenarios.

Summary updated

BibTeX
@inproceedings{chen2025skeleton,
  author = {Haipeng Chen and Yuheng Yang and Yingda Lyu},
  title = {{Skeleton-based Action Recognition with Non-linear Dependency Modeling and Hilbert-Schmidt Independence Criterion}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i2.32201},
}
Figure