Abstract
Distilling the complex spatial-temporal knowledge of heavy 3D-CNNs into lightweight models is a major challenge for mobile action recognition. This paper introduces a novel generative framework that uses Conditional Variational Autoencoders (CVAEs) to capture and transfer rich feature semantics, significantly boosting the performance of compact student models across multiple benchmarks.
Introduction
Action recognition is a cornerstone of intelligent video analysis, but state-of-the-art 3D-CNN models are computationally expensive, hindering deployment on resource-constrained edge devices. While Knowledge Distillation (KD) is a standard solution, existing methods often focus on simple loss-function matching or cross-modal fusion, neglecting the underlying spatial-temporal feature semantics—the "how" and "where" of motion. This paper addresses this gap by proposing a generative approach to transfer the distribution and logic of the teacher's feature space.
Methodology
The proposed framework operates in two distinct stages to ensure deep semantic transfer:
- Feature Representation: An attention module is integrated to highlight critical feature semantics. A Conditional Variational Autoencoder (CVAE) is then trained to learn the distribution of these features, conditioned on the generated attention maps. This allows the CVAE to effectively "reconstruct" the teacher's semantic understanding.
- Generative-based Feature Distillation: The student model is trained to mimic the teacher through two sub-processes:
- Generative Distillation: The student utilizes the pre-trained CVAE to ensure its features can reconstruct high-quality semantic information identical to the teacher's.
- Attention Distillation: The student is forced to align its attention maps with the teacher's, ensuring it focuses on the same spatial-temporal regions for action identification.
By treating distillation as a generative problem rather than simple regression, the framework bridges the gap between complex teacher architectures and efficient student models.
Experimental Evaluation
The framework was evaluated on three benchmarks: UCF101, HMDB51, and THUMOS14 (for action detection).
Key Results
The method consistently outperformed existing KD techniques across various metrics:
- Accuracy Improvements: On UCF101 and HMDB51, the student model achieved a Top-1 accuracy boost of 2.5%.
- Action Detection: On THUMOS14, the average mean Average Precision (mAP) improved by 1.1%, demonstrating efficacy in localization tasks.
- Cross-Architecture Distillation: A significant achievement was distilling knowledge from a 3D-CNN teacher to a 2D-CNN student (I2D), resulting in a 6.4% accuracy increase.
- Efficiency: The student models achieved these gains while utilizing only approximately 40.9% of the teacher's FLOPs.
Conclusion
This research demonstrates that matching semantic distributions via generative models is superior to raw pixel-wise feature matching for action recognition distillation. While the two-stage training and CVAE integration add complexity to the training phase, the resulting student models are highly efficient and performant for real-time inference. Future work may extend this generative approach to more complex video understanding tasks.
Summary updated