Self-Distillation for Further Pre-training of Transformers
TL;DR
Adapting large pre-trained Transformers to specific domains often leads to overfitting, especially in computer vision. This paper introduces a self-distillation strategy for the "further pre-training" stage. By using a domain-adapted model as a teacher to guide a student model, researchers can regularize the adaptation process, leading to significantly better generalization on downstream image and text classification tasks.
What Problem Does This Paper Address?
The standard "pre-train then fine-tune" pipeline works wonders when the pre-training data and the downstream task data are similar. However, it hits a wall when there is a significant "domain gap." For example, a model pre-trained on general internet images might struggle to interpret specialized medical X-rays.
To bridge this gap, researchers often use further pre-training—continuing self-supervised learning on unlabeled data from the target domain before fine-tuning. While this works well for Language Models, the authors found that Vision Transformers (ViTs) are highly susceptible to overfitting during this stage. Instead of learning useful domain features, the model "forgets" general representations, leading to worse performance than if they hadn't done the extra pre-training at all.
Methods
The authors propose a three-stage adaptation process called Self-Distillation for Further Pre-training:
- Teacher Preparation: Take an initial pre-trained model and perform standard further pre-training (using Masked Auto-Encoding, or MAE) on the target domain's unlabeled data. This model becomes the "Teacher."
- Self-Distillation: Take a fresh copy of the original pre-trained model to serve as the "Student." Train this student on the target unlabeled data using a dual objective:
- MAE Loss: The standard task of reconstructing masked patches/tokens.
- Distillation Loss: A regularization term that forces the student's hidden representations to stay close to the teacher's representations.
- Fine-tuning: Finally, fine-tune the distilled student on the small amount of labeled data available for the specific task.
The key novelty is using self-distillation not for model compression, but as a regularizer to prevent the model from drifting too far into an overfitted state during domain adaptation.
Datasets
The researchers evaluated their method across a wide variety of specialized domains:
- Image Classification: FGVC Aircraft, Caltech UCSD Birds (CUB), Chest X-ray (pneumonia diagnosis), Describable Textures (DTD), Stanford Dogs, and Oxford 102 Flowers.
- Text Classification: Chemprot (chemical-protein interactions), ACL-ARC (citation intent), SCIERC (scientific relations), and Twitter-Emotion.
- Low-Resource Analysis: Subsets of the CIFAR-100 dataset were used to test how the model performs with extremely limited labeled data.
Key Insights and Ideas
- The ViT Overfitting Trap: Naive further pre-training on a small domain-specific dataset causes ViTs to lose their generalization power.
As shown above, while training loss might drop, test accuracy on downstream tasks actually plummets without regularization. Self-distillation effectively stabilizes this curve.
- Regularization via Representation: By forcing the student to mimic the teacher's hidden layers, the model retains the "knowledge" from the massive general pre-training while still absorbing the nuances of the new domain.
- Diminishing Returns of Rounds: Theoretically and empirically, the authors show that the first round of self-distillation provides the most significant boost. Adding more rounds of distillation (where the Student becomes the Teacher for the next Student) offers only marginal gains at a high computational cost.
- Weight Distance Matters: The paper provides a theoretical analysis suggesting that self-distillation works by controlling the distance between the initial pre-trained weights and the final fine-tuned weights, which is a known factor in generalization.
Key Results
The proposed method consistently outperformed standard fine-tuning and other popular regularization techniques (like RecAdam or MARS):
- Vision Tasks: On the CUB (Birds) dataset, accuracy improved from 55.55% (standard fine-tuning) to 58.06%. On Chest X-rays, it jumped from 77.15% to 79.68%.
- Text Tasks: On the ACL-ARC dataset, the F1 score rose from 64.09 to 73.17, a massive improvement over naive further pre-training.
- Low-Data Hero: The method shines brightest when labeled data is scarce. On a 1,000-sample subset of CIFAR-100, self-distillation provided a 13% absolute accuracy boost over simple fine-tuning.
Limitations and Future Work
- Computational Overhead: The method requires training the model twice on the unlabeled data (once for the teacher, once for the student), which effectively doubles the pre-training time.
- Scope of Tasks: The evaluation focused primarily on classification. It remains to be seen how this regularization affects dense prediction tasks like object detection or segmentation, or generative tasks.
- Simplified Theory: While the authors provide a theoretical proof, it relies on a simplified linear model. Extending this theory to the complex, non-linear dynamics of full-scale Transformers is an open challenge.
Summary updated