Self-Distillation for Further Pre-training of Transformers

Seanie Lee, Minki Kang, Juho Lee, Sung Ju Hwang, Kenji Kawaguchi

Self-Distillation for Further Pre-training of Transformers

TL;DR

Adapting large pre-trained Transformers to specific domains often leads to overfitting, especially in computer vision. This paper introduces a self-distillation strategy for the "further pre-training" stage. By using a domain-adapted model as a teacher to guide a student model, researchers can regularize the adaptation process, leading to significantly better generalization on downstream image and text classification tasks.

What Problem Does This Paper Address?

The standard "pre-train then fine-tune" pipeline works wonders when the pre-training data and the downstream task data are similar. However, it hits a wall when there is a significant "domain gap." For example, a model pre-trained on general internet images might struggle to interpret specialized medical X-rays.

To bridge this gap, researchers often use further pre-training—continuing self-supervised learning on unlabeled data from the target domain before fine-tuning. While this works well for Language Models, the authors found that Vision Transformers (ViTs) are highly susceptible to overfitting during this stage. Instead of learning useful domain features, the model "forgets" general representations, leading to worse performance than if they hadn't done the extra pre-training at all.

Methods

The authors propose a three-stage adaptation process called Self-Distillation for Further Pre-training:

Methodology Overview
  1. Teacher Preparation: Take an initial pre-trained model and perform standard further pre-training (using Masked Auto-Encoding, or MAE) on the target domain's unlabeled data. This model becomes the "Teacher."
  2. Self-Distillation: Take a fresh copy of the original pre-trained model to serve as the "Student." Train this student on the target unlabeled data using a dual objective:
  3. MAE Loss: The standard task of reconstructing masked patches/tokens.
  4. Distillation Loss: A regularization term that forces the student's hidden representations to stay close to the teacher's representations.
  5. Fine-tuning: Finally, fine-tune the distilled student on the small amount of labeled data available for the specific task.

The key novelty is using self-distillation not for model compression, but as a regularizer to prevent the model from drifting too far into an overfitted state during domain adaptation.

Datasets

The researchers evaluated their method across a wide variety of specialized domains:

  • Image Classification: FGVC Aircraft, Caltech UCSD Birds (CUB), Chest X-ray (pneumonia diagnosis), Describable Textures (DTD), Stanford Dogs, and Oxford 102 Flowers.
  • Text Classification: Chemprot (chemical-protein interactions), ACL-ARC (citation intent), SCIERC (scientific relations), and Twitter-Emotion.
  • Low-Resource Analysis: Subsets of the CIFAR-100 dataset were used to test how the model performs with extremely limited labeled data.

Key Insights and Ideas

  • The ViT Overfitting Trap: Naive further pre-training on a small domain-specific dataset causes ViTs to lose their generalization power.
Test Accuracy vs Further Pre-training Steps

As shown above, while training loss might drop, test accuracy on downstream tasks actually plummets without regularization. Self-distillation effectively stabilizes this curve.

  • Regularization via Representation: By forcing the student to mimic the teacher's hidden layers, the model retains the "knowledge" from the massive general pre-training while still absorbing the nuances of the new domain.
  • Diminishing Returns of Rounds: Theoretically and empirically, the authors show that the first round of self-distillation provides the most significant boost. Adding more rounds of distillation (where the Student becomes the Teacher for the next Student) offers only marginal gains at a high computational cost.
  • Weight Distance Matters: The paper provides a theoretical analysis suggesting that self-distillation works by controlling the distance between the initial pre-trained weights and the final fine-tuned weights, which is a known factor in generalization.

Key Results

The proposed method consistently outperformed standard fine-tuning and other popular regularization techniques (like RecAdam or MARS):

  • Vision Tasks: On the CUB (Birds) dataset, accuracy improved from 55.55% (standard fine-tuning) to 58.06%. On Chest X-rays, it jumped from 77.15% to 79.68%.
  • Text Tasks: On the ACL-ARC dataset, the F1 score rose from 64.09 to 73.17, a massive improvement over naive further pre-training.
  • Low-Data Hero: The method shines brightest when labeled data is scarce. On a 1,000-sample subset of CIFAR-100, self-distillation provided a 13% absolute accuracy boost over simple fine-tuning.

Limitations and Future Work

  • Computational Overhead: The method requires training the model twice on the unlabeled data (once for the teacher, once for the student), which effectively doubles the pre-training time.
  • Scope of Tasks: The evaluation focused primarily on classification. It remains to be seen how this regularization affects dense prediction tasks like object detection or segmentation, or generative tasks.
  • Simplified Theory: While the authors provide a theoretical proof, it relies on a simplified linear model. Extending this theory to the complex, non-linear dynamics of full-scale Transformers is an open challenge.

Summary updated

Abstract

Pre-training a large transformer model on a massive amount of unlabeled data and fine-tuning it on labeled datasets for diverse downstream tasks has proven to be a successful strategy, for a variety of vision and natural language processing tasks. However, direct fine-tuning of the pre-trained model may be suboptimal if there exist large discrepancies across data domains for pre-training and fine-tuning. To tackle this issue, several previous studies have proposed further pre-training strategies, where we continue to pre-train the model on the target unlabeled dataset before fine-tuning. However, all of them solely focus on language models and we empirically find that a Vision Transformer is vulnerable to overfitting as we continue to pretrain the model on target unlabeled data. In order to tackle this limitation, we propose self-distillation as a regularization for a further pre-training stage. Specifically, we first further pre-train the initial pre-trained model on the target unlabeled data and then consider it as a teacher for self-distillation. Then we take the same initial pre-trained model as a student and enforce its hidden representations to be close to those of the teacher while optimizing the student with a masked auto-encoding objective. We empirically validate the efficacy of self-distillation on a variety of benchmark datasets for image and text classification tasks. Experimentally, we show that our proposed method outperforms all the relevant baselines. Theoretically, we analyze the proposed method with a simplified model to understand how self-distillation for further pre-training can potentially help improve the performance of the downstream tasks.
BibTeX
@inproceedings{lee2023self,
  author = {Seanie Lee and Minki Kang and Juho Lee and Sung Ju Hwang and Kenji Kawaguchi},
  title = {{Self-Distillation for Further Pre-training of Transformers}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=kj6oK_Hj40},
}
Figure