Action recognition using ensembling of different distillation-trained spatial-temporal graph convolution models

Qingshan Yang, Tai‐Jiang MuJournal of Image and Graphics 2022 · DOI

Abstract

Deep learning has significantly advanced action recognition, yet challenges remain regarding model robustness and computational complexity. Current state-of-the-art models often struggle with large-scale datasets and diverse viewpoints while requiring substantial parameters. This paper presents a lightweight spatial-temporal graph convolutional ensemble model trained via knowledge distillation. By redesigning standard graph convolution layers with group convolutions and employing a multi-teacher distillation strategy, the proposed model achieves superior performance on the NTU RGB+D dataset, outperforming its larger teacher models.

Introduction

Skeleton-based action recognition is a critical task in computer vision, with applications ranging from smart surveillance to human-computer interaction. While Spatial-Temporal Graph Convolutional Networks (ST-GCNs) have become the standard for processing human joint data, they often face a trade-off between accuracy and efficiency. High-performing models typically involve complex architectures with massive parameter counts, making them difficult to deploy in real-time or resource-constrained environments. Furthermore, these models can be sensitive to viewpoint variations.

This research addresses these challenges by proposing a lightweight ensemble framework. The goal is to create a model that is computationally efficient without sacrificing—and ideally improving—the recognition accuracy and robustness of existing heavy-duty architectures.

Methodology

The proposed approach integrates three main components: lightweight architectural design, multi-teacher knowledge distillation, and multi-stream ensembling.

Lightweight Sub-network Design

The core of the framework consists of redesigned graph convolution modules: Light-DGC (Directed Graph Convolution) and Light-AGC (Adaptive Graph Convolution). To reduce the parameter footprint, the authors replaced standard temporal convolutions with 2D depth-wise group convolutions. This design choice significantly slashes the number of parameters in the temporal dimension while maintaining a large receptive field, allowing the model to capture long-range temporal dependencies efficiently.

Multi-Teacher Knowledge Distillation (KD)

To train these lightweight "student" models, the researchers employed a knowledge distillation strategy. Two established high-performance models, 2s-AGCN (Two-Stream Adaptive Graph Convolutional Network) and DGNN (Directed Graph Neural Network), served as teachers.

The student models were trained using a composite loss function that balances:

  1. Hard Targets: Standard cross-entropy loss against the ground-truth action labels.
  2. Soft Targets: Distillation loss that encourages the student to mimic the output probability distributions and feature representations of the teacher models.

This strategy allows the lightweight students to inherit the complex feature-extraction capabilities of the larger teachers.

Data Augmentation and Ensemble Strategy

To enhance robustness against viewpoint changes, the researchers applied affine transformations to the skeleton data during training. The final recognition system is an ensemble of six sub-networks. These sub-networks process different input modalities:

  • Original joint coordinates
  • Bone lengths and orientations
  • Affine-transformed versions of both joints and bones

The final prediction is obtained through a linear fusion of the outputs from these six distilled sub-networks.

Experimental Results

The model was rigorously evaluated on the NTU RGB+D dataset, a benchmark containing over 56,000 samples across 60 action classes. Performance was measured using the Cross-Subject (CS) and Cross-View (CV) protocols.

The ensemble model achieved an accuracy of 90.9% on CS and 96.5% on CV. Notably, the distilled student ensemble outperformed its individual teachers:

  • vs. 2s-AGCN: +2.4% (CS) and +1.4% (CV) improvement.
  • vs. DGNN: +1.0% (CS) and +0.4% (CV) improvement.
  • vs. MS-AAGCN: +0.9% (CS) and +0.3% (CV) improvement.

These results demonstrate that the distillation process, combined with ensembling, acts as an effective regularizer, allowing the lightweight ensemble to capture a more diverse and robust set of features than a single large-scale model.

Conclusion

This study demonstrates that lightweight graph convolutional models, when trained through multi-teacher distillation and combined in an ensemble, can surpass the performance of state-of-the-art heavy models. The use of group convolutions provides a path toward efficient real-world deployment, while the multi-stream approach ensures robustness across different subjects and camera viewpoints. Future work will explore integrating additional modalities, such as RGB context, to improve performance on fine-grained actions where skeleton data alone may be insufficient.

Summary updated

Abstract

目的基于深度学习的动作识别方法识别准确率显著提升,但仍然存在很多挑战和困难。现行方法在一些训练数据大、分类类别多的数据集以及实际应用中鲁棒性较差,而且许多方法使用的模型参数量较大、计算复杂,提高模型准确度和鲁棒性的同时对模型进行轻量化仍然是一个重要的研究方向。为此,提出了一种基于知识蒸馏的轻量化时空图卷积动作识别融合模型。方法改进最新的时空卷积网络,利用分组卷积等设计参数量较少的时空卷积子模型; 为了训练该模型,选取两个现有的基于全卷积的模型作为教师模型在数据集上训练,在得到训练好的教师模型后,再利用知识蒸馏的方法结合数据增强技术训练参数量较少的时空卷积子模型; 利用线性融合的方法将知识蒸馏训练得到的子模型融合得到最终的融合模型。结果在广泛使用的NTU RGB + D数据集上与前沿的多种方法进行了比较,在CS(cross-subject)和CV(cross-view)两种评估标准下, 本文模型的准确率分别为90.9 %和96.5 %,与教师模型2s-AGCN(two-stream adaptive graph convolutional networks for skeleton-based action)相比,分别提高了2.4 %和1.4 %; 与教师模型DGNN(directed graph neural network)相比,分别提高了1.0 %和0.4 %; 与MS-AAGCN(multi-stream attention-enhanced adaptive graph convolutional neural network)模型相比,分别提高了0.9 %和0.3 %。结论本文提出的融合模型,综合了知识蒸馏、数据增强技术和模型融合的优点,使动作识别更加准确和鲁棒。
BibTeX
@article{yang2022action,
  author = {Qingshan Yang and Tai‐Jiang Mu},
  title = {{Action recognition using ensembling of different distillation-trained spatial-temporal graph convolution models}},
  journal = {Journal of Image and Graphics},
  year = {2022},
  doi = {10.11834/jig.200791},
}
Figure