Weight Copy and Low-Rank Adaptation for Few-Shot Distillation of Vision Transformers

Diana-Nicoleta Grigore, Mariana-Iuliana Georgescu, Jon Alvarez Justo, Tor Johansen, Andreea Iuliana Ionescu, Radu Tudor IonescuWACV 2025

TL;DR

This paper introduces WeCoLoRA, a novel method for few-shot knowledge distillation in vision transformers. By combining weight copying from intermittent teacher layers with an enhanced Low-Rank Adaptation (LoRA) technique, WeCoLoRA effectively trains smaller student models on limited data, achieving state-of-the-art performance across various domains.

What Problem Does This Paper Address?

Vision transformers (ViTs) have achieved remarkable success but often require massive datasets and computational resources for training, hindering their deployment on edge devices. While knowledge distillation (KD) offers a solution by transferring knowledge from large "teacher" models to smaller "student" models, few-shot KD for ViTs, especially during the pre-training stage, remains underexplored. Existing KD methods struggle with the inherent overfitting tendencies of ViTs on small datasets and the computational demands of adapting large models. This paper aims to bridge this gap by proposing an efficient and effective few-shot KD approach for ViTs that significantly reduces data and computational requirements.

Methods

The proposed method, WeCoLoRA, operates in two main steps:

  1. Intermittent Weight Copying: Leveraging the consistent depth-wise structure of ViTs, WeCoLoRA copies weights from specific, intermittent layers of a pre-trained teacher ViT directly into a shallower student ViT. An "intermittence factor" controls the compression ratio between the teacher and student, determining how many layers are skipped. This step initializes the student with a strong structural foundation.
  2. Enhanced Low-Rank Adaptation (LoRA): After weight copying, an enhanced LoRA technique is employed to distill knowledge into the student, specifically to recover the information processing capabilities of the skipped teacher layers. Unlike standard LoRA which typically adapts query and value projection matrices, WeCoLoRA's enhanced version applies LoRA to all components of the transformer blocks (query, key, value projections, and feed-forward networks). This allows for a more faithful replication of the teacher's behavior in the adapted layers. The enhanced LoRA parameters are trained via feature distillation on unlabeled images.

Datasets

The paper uses the following datasets:

  • ImageNet-1K: Used for knowledge distillation in few-shot settings (0.25% to 10% of the original training set) and for linear probing evaluation.
  • CIFAR-100: Used for linear probing evaluation.
  • ChestX-ray14: Used for linear probing evaluation (multi-label classification).
  • iNaturalist: Used for linear probing evaluation.
  • RESISC45: Used for linear probing evaluation.

The paper does not introduce new benchmarks but evaluates on existing diverse datasets to demonstrate the generalizability of their approach.

Key Insights and Ideas

  • Weight copying is a powerful initialization: Directly copying weights from intermittent teacher layers provides a strong starting point for the student, significantly boosting performance compared to training from scratch.
  • Enhanced LoRA is crucial for recovering skipped knowledge: Standard LoRA is insufficient to fully capture the functionality of skipped transformer blocks. Applying LoRA to all components of the transformer block is key to effectively learning the distilled knowledge.
  • Few-shot KD can be effective even with limited data: The proposed method demonstrates that significant knowledge can be transferred with as little as 1% of the original training data, making it suitable for data-scarce scenarios.
  • The intermittence factor (r) controls complexity and performance: This parameter allows for a trade-off between model size, inference speed, and performance, enabling customization for different resource constraints.
  • LoRA's rank (k) is a hyperparameter for bias-variance trade-off: Tuning the rank of LoRA matrices can help balance model complexity and prevent overfitting on small datasets.

Key Results

  • State-of-the-art performance: WeCoLoRA consistently outperforms existing state-of-the-art few-shot KD methods (DeiT, MiniViT, DMAE, DoRA) across multiple datasets and teacher types (supervised and self-supervised).
  • Significant gains with few-shot learning: On ImageNet-1K with only 1% of the training data, WeCoLoRA achieved an accuracy of 60.5%, a 25% improvement over its ablated version (WeCo+KD).
  • Efficiency: The student model trained with WeCoLoRA is twice as fast as the teacher model.
  • Reduced trainable parameters: WeCoLoRA adds significantly fewer trainable parameters (10M with k=128) compared to its ablated version (44M).
  • Robust features: Latent space visualizations (t-SNE) show that WeCoLoRA learns more discriminative and disentangled features, leading to better performance on downstream tasks. Attention visualizations also indicate a focus on more discriminative features.

Limitations and Future Work

  • Hyperparameter tuning: The performance of WeCoLoRA is sensitive to hyperparameters like the intermittence factor (r) and LoRA rank (k), requiring careful tuning for optimal results.
  • Computational evaluation: While the paper reports running times and GFLOPs, further extensive computational analysis across different hardware could be beneficial.
  • Future work: The authors suggest exploring WeCoLoRA on other architectures and tasks, as well as investigating more advanced distillation techniques to further improve performance. The paper also implies that the framework can be extended to other domains beyond computer vision.

Summary updated

BibTeX
@inproceedings{grigore2025weight,
  author = {Diana-Nicoleta Grigore and Mariana-Iuliana Georgescu and Jon Alvarez Justo and Tor Johansen and Andreea Iuliana Ionescu and Radu Tudor Ionescu},
  title = {{Weight Copy and Low-Rank Adaptation for Few-Shot Distillation of Vision Transformers}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure