Abstract
Deep learning has significantly advanced action recognition, yet challenges remain regarding model robustness and computational complexity. Current state-of-the-art models often struggle with large-scale datasets and diverse viewpoints while requiring substantial parameters. This paper presents a lightweight spatial-temporal graph convolutional ensemble model trained via knowledge distillation. By redesigning standard graph convolution layers with group convolutions and employing a multi-teacher distillation strategy, the proposed model achieves superior performance on the NTU RGB+D dataset, outperforming its larger teacher models.
Introduction
Skeleton-based action recognition is a critical task in computer vision, with applications ranging from smart surveillance to human-computer interaction. While Spatial-Temporal Graph Convolutional Networks (ST-GCNs) have become the standard for processing human joint data, they often face a trade-off between accuracy and efficiency. High-performing models typically involve complex architectures with massive parameter counts, making them difficult to deploy in real-time or resource-constrained environments. Furthermore, these models can be sensitive to viewpoint variations.
This research addresses these challenges by proposing a lightweight ensemble framework. The goal is to create a model that is computationally efficient without sacrificing—and ideally improving—the recognition accuracy and robustness of existing heavy-duty architectures.
Methodology
The proposed approach integrates three main components: lightweight architectural design, multi-teacher knowledge distillation, and multi-stream ensembling.
Lightweight Sub-network Design
The core of the framework consists of redesigned graph convolution modules: Light-DGC (Directed Graph Convolution) and Light-AGC (Adaptive Graph Convolution). To reduce the parameter footprint, the authors replaced standard temporal convolutions with 2D depth-wise group convolutions. This design choice significantly slashes the number of parameters in the temporal dimension while maintaining a large receptive field, allowing the model to capture long-range temporal dependencies efficiently.
Multi-Teacher Knowledge Distillation (KD)
To train these lightweight "student" models, the researchers employed a knowledge distillation strategy. Two established high-performance models, 2s-AGCN (Two-Stream Adaptive Graph Convolutional Network) and DGNN (Directed Graph Neural Network), served as teachers.
The student models were trained using a composite loss function that balances:
- Hard Targets: Standard cross-entropy loss against the ground-truth action labels.
- Soft Targets: Distillation loss that encourages the student to mimic the output probability distributions and feature representations of the teacher models.
This strategy allows the lightweight students to inherit the complex feature-extraction capabilities of the larger teachers.
Data Augmentation and Ensemble Strategy
To enhance robustness against viewpoint changes, the researchers applied affine transformations to the skeleton data during training. The final recognition system is an ensemble of six sub-networks. These sub-networks process different input modalities:
- Original joint coordinates
- Bone lengths and orientations
- Affine-transformed versions of both joints and bones
The final prediction is obtained through a linear fusion of the outputs from these six distilled sub-networks.
Experimental Results
The model was rigorously evaluated on the NTU RGB+D dataset, a benchmark containing over 56,000 samples across 60 action classes. Performance was measured using the Cross-Subject (CS) and Cross-View (CV) protocols.
The ensemble model achieved an accuracy of 90.9% on CS and 96.5% on CV. Notably, the distilled student ensemble outperformed its individual teachers:
- vs. 2s-AGCN: +2.4% (CS) and +1.4% (CV) improvement.
- vs. DGNN: +1.0% (CS) and +0.4% (CV) improvement.
- vs. MS-AAGCN: +0.9% (CS) and +0.3% (CV) improvement.
These results demonstrate that the distillation process, combined with ensembling, acts as an effective regularizer, allowing the lightweight ensemble to capture a more diverse and robust set of features than a single large-scale model.
Conclusion
This study demonstrates that lightweight graph convolutional models, when trained through multi-teacher distillation and combined in an ensemble, can surpass the performance of state-of-the-art heavy models. The use of group convolutions provides a path toward efficient real-world deployment, while the multi-stream approach ensures robustness across different subjects and camera viewpoints. Future work will explore integrating additional modalities, such as RGB context, to improve performance on fine-grained actions where skeleton data alone may be insufficient.
Summary updated