Self-Distillation for Further Pre-training of Transformers

Seanie Lee, Minki Kang, Juho Lee, Sung Ju Hwang, Kenji Kawaguchi

Abstract

Pre-training a large transformer model on a massive amount of unlabeled data and fine-tuning it on labeled datasets for diverse downstream tasks has proven to be a successful strategy, for a variety of vision and natural language processing tasks. However, direct fine-tuning of the pre-trained model may be suboptimal if there exist large discrepancies across data domains for pre-training and fine-tuning. To tackle this issue, several previous studies have proposed further pre-training strategies, where we continue to pre-train the model on the target unlabeled dataset before fine-tuning. However, all of them solely focus on language models and we empirically find that a Vision Transformer is vulnerable to overfitting as we continue to pretrain the model on target unlabeled data. In order to tackle this limitation, we propose self-distillation as a regularization for a further pre-training stage. Specifically, we first further pre-train the initial pre-trained model on the target unlabeled data and then consider it as a teacher for self-distillation. Then we take the same initial pre-trained model as a student and enforce its hidden representations to be close to those of the teacher while optimizing the student with a masked auto-encoding objective. We empirically validate the efficacy of self-distillation on a variety of benchmark datasets for image and text classification tasks. Experimentally, we show that our proposed method outperforms all the relevant baselines. Theoretically, we analyze the proposed method with a simplified model to understand how self-distillation for further pre-training can potentially help improve the performance of the downstream tasks.

Introduction

Large Transformer models, such as BERT for natural language processing and ViT for computer vision, have revolutionized machine learning by enabling a "pre-train then fine-tune" paradigm. Typically, these models are pre-trained on vast, general-purpose datasets (e.g., ImageNet or Wikipedia) and subsequently fine-tuned on specific downstream tasks. However, a significant challenge arises when the downstream task belongs to a domain far removed from the pre-training data—such as specialized medical imaging or scientific literature. This "distributional discrepancy" often leads to suboptimal performance.

To bridge this gap, practitioners often employ further pre-training: continuing self-supervised training on unlabeled data from the target domain before fine-tuning. While effective for language models, this paper identifies a critical failure mode in Vision Transformers (ViTs): they are highly susceptible to overfitting during this intermediate stage. Instead of acquiring useful domain-specific features, the model often "forgets" its general knowledge, leading to degraded generalization on the final labeled task.

Methodology

To address the overfitting observed in naive further pre-training, we propose a Self-Distillation regularization framework. The core idea is to use a domain-adapted version of the model to guide a fresh copy of the original pre-trained model, preventing it from drifting into an overfitted state.

The proposed pipeline consists of three distinct stages:

  1. Teacher Creation: An initial pre-trained model undergoes "naive" further pre-training on the target unlabeled dataset using a Masked Auto-Encoding (MAE) objective. This model, which may be slightly overfitted but has seen the target domain, serves as the Teacher.
  2. Student Training (Self-Distillation): A fresh copy of the original pre-trained model (the Student) is trained on the same target unlabeled data. The optimization uses a dual objective: a standard MAE Loss for reconstruction and a Distillation Loss that penalizes the distance between the Student’s and Teacher’s hidden representations.
  3. Fine-tuning: The distilled Student model is then fine-tuned on the labeled downstream dataset.
Comparison of standard fine-tuning, naive further pre-training, and the proposed self-distillation framework.

This distillation loss acts as a "tether," allowing the student to assimilate domain-specific features from the teacher while maintaining the robust general representations learned during initial pre-training.

Experimental Evaluation

The efficacy of self-distillation was evaluated across multiple modalities and diverse datasets:

  • Computer Vision: Six datasets including FGVC Aircraft, CUB (Birds), Chest X-ray, DTD (Textures), Stanford Dogs, and Oxford Flowers.
  • Natural Language Processing: Four datasets including SCIERC (Scientific abstracts), ACL-ARC (Citation intent), Chemprot (Chemical-protein interactions), and Twitter-Emotion.
  • Low-Resource Scenarios: CIFAR-100 was used to evaluate performance when labeled data is extremely limited.

Results and Analysis

The experimental results consistently demonstrate that self-distillation outperforms both standard fine-tuning and naive further pre-training.

The Overfitting Paradox

A key finding is that in ViTs, naive further pre-training often leads to a decrease in test accuracy despite a continued decrease in training loss on the unlabeled data. As shown in Figure 1, while naive pre-training performance peaks early and then sharply declines (indicating overfitting), the self-distillation approach maintains higher accuracy and greater stability over more training steps.

Test accuracy on the CUB dataset as a function of further pre-training steps, highlighting the overfitting of naive pre-training versus the stability of self-distillation.

Quantitative Gains

  • Vision: On the Chest X-ray dataset, self-distillation achieved 79.68% accuracy, a significant improvement over the 77.15% achieved by standard fine-tuning. On the CUB dataset, accuracy rose from 55.55% to 58.06%.
  • NLP: The method showed substantial gains in specialized text domains. For instance, on ACL-ARC, the F1 score jumped from 64.09% (fine-tuning) to 73.17%.
  • Low-Data Efficiency: In a low-resource setting with only 1,000 labeled examples from CIFAR-100, self-distillation provided a 13% absolute improvement over simple fine-tuning.

Discussion and Conclusion

Beyond empirical success, the authors provide theoretical analysis showing that self-distillation strictly decreases the generalization bound. By minimizing the distance between the initial pre-trained weights and the final fine-tuned weights—a known proxy for generalization—the method ensures more robust adaptation.

Self-distillation offers a powerful, task-agnostic regularization strategy for adapting Transformers to new domains. While the approach introduces additional computational overhead—increasing training time significantly compared to simple fine-tuning—the substantial gains in accuracy and robustness make it a compelling choice for domain-specific applications. Future work may focus on optimizing the distillation process to reduce the "wall-clock" time required for deployment.

Summary updated

Abstract

Pre-training a large transformer model on a massive amount of unlabeled data and fine-tuning it on labeled datasets for diverse downstream tasks has proven to be a successful strategy, for a variety of vision and natural language processing tasks. However, direct fine-tuning of the pre-trained model may be suboptimal if there exist large discrepancies across data domains for pre-training and fine-tuning. To tackle this issue, several previous studies have proposed further pre-training strategies, where we continue to pre-train the model on the target unlabeled dataset before fine-tuning. However, all of them solely focus on language models and we empirically find that a Vision Transformer is vulnerable to overfitting as we continue to pretrain the model on target unlabeled data. In order to tackle this limitation, we propose self-distillation as a regularization for a further pre-training stage. Specifically, we first further pre-train the initial pre-trained model on the target unlabeled data and then consider it as a teacher for self-distillation. Then we take the same initial pre-trained model as a student and enforce its hidden representations to be close to those of the teacher while optimizing the student with a masked auto-encoding objective. We empirically validate the efficacy of self-distillation on a variety of benchmark datasets for image and text classification tasks. Experimentally, we show that our proposed method outperforms all the relevant baselines. Theoretically, we analyze the proposed method with a simplified model to understand how self-distillation for further pre-training can potentially help improve the performance of the downstream tasks.
BibTeX
@inproceedings{lee2023self,
  author = {Seanie Lee and Minki Kang and Juho Lee and Sung Ju Hwang and Kenji Kawaguchi},
  title = {{Self-Distillation for Further Pre-training of Transformers}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=kj6oK_Hj40},
}
Figure