Preserve Pre-trained Knowledge: Transfer Learning With Self-Distillation For Action Recognition

Yang Zhou, Zhanhao He, Keyu Lu, Guanhong Wang, Gaoang WangarXiv 2022 · OpenReview

Abstract

Video-based action recognition is one of the most popular topics in computer vision. With recent advances of self-supervised video representation learning approaches, action recognition usually follows a two-stage training framework, i.e., self-supervised pre-training on large-scale unlabeled sets and transfer learning on a downstream labeled set. However, catastrophic forgetting of the pre-trained knowledge becomes the main issue in the downstream transfer learning of action recognition, resulting in a sub-optimal solution. In this paper, to alleviate the above issue, we propose a novel transfer learning approach that combines self-distillation in fine-tuning to preserve knowledge from the pre-trained model learned from the large-scale dataset. Specifically, we fix the encoder from the last epoch as the teacher model to guide the training of the encoder from the current epoch in the transfer learning. With such a simple yet effective learning strategy, we outperform state-of-the-art methods on widely used UCF101 and HMDB51 datasets in action recognition task.

Introduction

In modern computer vision, the standard recipe for success is a two-stage framework: self-supervised pre-training on massive unlabeled datasets to learn general spatial-temporal features, followed by transfer learning (fine-tuning) on a smaller, labeled dataset for specific tasks. However, during the second stage, models often over-optimize for the small labeled set, leading to "catastrophic forgetting." This phenomenon causes the model to lose the robust, general representations learned during pre-training, resulting in sub-optimal generalization. This paper introduces a self-distillation regularization technique to ensure the model retains its pre-trained knowledge while learning new task-specific labels.

Methods

The proposed Self-Distillation Guided Transfer Learning approach treats the model from the previous training epoch as a "teacher" for the model in the current epoch. This acts as a temporal anchor, preventing the weights from drifting too far from the pre-trained state.

Overview of the self-distillation guided transfer learning framework.

The Mechanism

At the start of every training epoch, a copy of the spatial-temporal encoder from the previous epoch is made and its parameters are frozen. This copy serves as the teacher network $f'$. The current model (the student $f$) is then optimized using a dual-loss objective:

  1. Cross-Entropy Loss ($L_{CE}$): The standard classification loss based on ground-truth labels.
  2. Self-Distillation Loss ($L_{SD}$): A regularization term calculating the Euclidean distance between the feature representations produced by the current encoder $f(x)$ and the teacher encoder $f'(x)$.

A hyperparameter $\lambda$ balances these losses. This architecture-agnostic approach can be integrated into various backbones, including 3D CNNs and R(2+1)D networks, without modifying the underlying model structure.

Experimental Results

The method was evaluated on two primary action recognition benchmarks: UCF101 (101 categories) and HMDB51 (51 categories). Pre-training was conducted using RSPNet, a state-of-the-art self-supervised model.

Comparison of the proposed method against state-of-the-art approaches on UCF101 and HMDB51.

As shown in the results, the proposed method consistently outperforms the baseline (RSPNet) and other state-of-the-art methods across different architectures:

  • UCF101: Achieved 82.3% Top-1 accuracy with the R(2+1)D architecture, a significant improvement over the 80.2% baseline.
  • HMDB51: Achieved 47.3% Top-1 accuracy, compared to the 44.1% baseline.
  • Robustness: Ablation studies indicated that the method is robust to the choice of $\lambda$, outperforming the baseline across a wide range of values (from 0.1 to 1000).

Conclusion

By constraining the Euclidean distance between current and past representations, self-distillation effectively preserves the spatial-temporal features learned during pre-training. This simple yet powerful strategy demonstrates that "remembering" pre-trained knowledge is critical for successful transfer learning in action recognition. Future work may explore optimizing the computational overhead of the dual-forward pass and testing the method's efficacy in other domains like NLP or 2D image classification.

Summary updated

Abstract

Video-based action recognition is one of the most popular topics in computer vision. With recent advances of selfsupervised video representation learning approaches, action recognition usually follows a two-stage training framework, i.e., self-supervised pre-training on large-scale unlabeled sets and transfer learning on a downstream labeled set. However, catastrophic forgetting of the pre-trained knowledge becomes the main issue in the downstream transfer learning of action recognition, resulting in a sub-optimal solution. In this paper, to alleviate the above issue, we propose a novel transfer learning approach that combines self-distillation in fine-tuning to preserve knowledge from the pre-trained model learned from the large-scale dataset. Specifically, we fix the encoder from the last epoch as the teacher model to guide the training of the encoder from the current epoch in the transfer learning. With such a simple yet effective learning strategy, we outperform state-of-the-art methods on widely used UCF101 and HMDB51 datasets in action recognition task.
BibTeX
@misc{zhou2022preserve,
  author = {Yang Zhou and Zhanhao He and Keyu Lu and Guanhong Wang and Gaoang Wang},
  title = {{Preserve Pre-trained Knowledge: Transfer Learning With Self-Distillation For Action Recognition}},
  year = {2022},
  eprint = {2205.00506},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.00506},
}
Figure