TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance

Kan Wu, Houwen Peng, Zhenghong Zhou, Bin Xiao, Mengchen Liu, Lu Yuan, Hong Xuan, Michael Valenzuela, Xi Chen, Xinggang Wang, Hongyang Chao, Han Hu

TL;DR

TinyCLIP is a novel cross-modal distillation framework that shrinks massive CLIP models by up to 90% while maintaining—or even exceeding—the original performance. By mimicking the "affinity" between vision and language and inheriting weights from larger teachers, TinyCLIP slashes training costs by up to 7.8× and enables high-performance, zero-shot models suitable for edge devices.

What Problem Does This Paper Address?

Large-scale models like CLIP (Contrastive Language-Image Pre-training) have revolutionized computer vision by enabling zero-shot transfer to almost any task. However, their massive size—often hundreds of millions of parameters—makes them prohibitively expensive to store, deploy on mobile devices, or even train from scratch (requiring thousands of GPU days).

While knowledge distillation is a standard way to shrink models, doing so for multi-modal models like CLIP presents unique hurdles:

  1. Cross-Modal Interaction: You aren't just distilling one encoder; you have to preserve the delicate alignment between the image and text branches.
  2. Computational Cost: Distilling on hundreds of millions of image-text pairs is computationally exhausting.
  3. Information Loss: Aggressive one-step compression often leads to "convergence failure," where the small model simply fails to learn the teacher's complex knowledge.

Methods

The authors introduce TinyCLIP, which relies on three primary pillars to achieve efficient compression:

1. Affinity Mimicking

Standard distillation often focuses on making the student's features look like the teacher's features. TinyCLIP instead focuses on the relationship between modalities. As shown in Figure 2, the student is trained to mimic the teacher's affinity matrix—the pairwise similarity scores between all images and text descriptions in a batch. This forces the student to learn not just what an image looks like, but how it relates to various textual concepts, including negative pairs.

Figure 2. Affinity mimicking for language-image models. The loss includes image-to-text loss (yellow) and text-to-image loss (blue).

2. Weight Inheritance

Rather than starting from random weights, TinyCLIP "inherits" weights from the pre-trained teacher. The authors propose two ways to select which weights to keep:

  • Manual Inheritance: Based on redundancy analysis, they select specific layers or channels.
  • Automatic Inheritance: They use learnable masks (Figure 3) and a sparsity constraint to automatically identify the most "important" weights for each modality during a preliminary pruning phase.
Figure 3. Weight inheritance of CLIP. Given a large pre-trained CLIP model, the inherited weights are selected by masks. The weights whose mask is 0 will be removed. For the manual way, the masks are set to 1 in the first k-dimension or uniform k-layer, the rest is set to 0. For the automatic way, the masks are learned jointly with the model under the sparsity constraint.

3. Multi-stage Progressive Distillation

To reach extreme compression ratios (e.g., >70% reduction) without crashing accuracy, TinyCLIP uses a "staircase" approach. It compresses the model in stages (e.g., 100% $\rightarrow$ 75% $\rightarrow$ 50%), allowing the student to stabilize at each level of sparsity before being squeezed further.

Datasets

The models were trained and evaluated on several large-scale benchmarks:

  • Training: LAION-400M (400 million image-text pairs) and YFCC-15M.
  • Evaluation: ImageNet-1K (for zero-shot and linear probe accuracy), Flickr30k, and MSCOCO (for image-text retrieval).
  • Downstream Tasks: A suite of 23 diverse datasets (e.g., Food101, CIFAR, SUN397) via the Elevater toolkit to test transferability.

Key Insights and Ideas

  • Asymmetric Redundancy: One of the most interesting findings is that the two encoders "waste" space differently. The text encoder is highly redundant in depth (you can remove layers), while the image encoder is redundant in width (you can remove channels/heads). Figure 4 illustrates this by showing how cosine similarity between layers behaves differently for each branch.
  • Teacher-Student Compatibility: A "stronger" teacher isn't always better. The authors found that weight inheritance works best when the teacher and student share similar architectures. A massive ViT-H/14 teacher actually performed worse for a ViT-B student than a ViT-B/32 teacher did.
  • Affinity > Features: Distilling the cross-modal affinity space is significantly more effective than distilling single-modality features, as it captures the "nuanced understanding" of how different concepts relate to one another.
Figure 4. The cosine similarity of the input and output embedding of each layer, and the zero-shot accuracy on ImageNet [7] after removing a layer for OpenCLIP ViT-B-32 [21, 46] pretrained on LAION-2B [49].

Key Results

TinyCLIP achieves state-of-the-art performance for small-scale CLIP models:

  • Efficiency: Distillation with weight inheritance sped up training by 1.4× to 7.8× compared to training from scratch.
  • Tiny but Mighty: Their TinyCLIP ViT-8M/16 (utilizing only 8.9% of the parameters of a standard ViT-B/16) achieved 41.1% zero-shot accuracy on ImageNet, actually beating the original CLIP ViT-B/16 by 3.5%.
  • Balanced Compression: A TinyCLIP-ViT at 50% size (63M params) achieved 61.4% accuracy, nearly matching the 62.9% of the full-sized OpenCLIP model while being 1.5× faster.
  • Robustness: TinyCLIP models maintained strong performance across out-of-distribution datasets like ImageNet-V2 and ImageNet-Sketch, proving the distillation process doesn't sacrifice the "generalization magic" of CLIP.

Limitations and Future Work

While TinyCLIP is highly effective, the authors note that extreme compression still requires multiple stages to remain stable, which adds some complexity to the pipeline. Additionally, while they explored ViT and ResNet architectures, the "automatic inheritance" masks were primarily tested on Transformer blocks.

Future work may involve exploring even more aggressive compression for "micro-models" and investigating how these techniques apply to the latest generation of even larger foundation models (like CLIP-L or ViT-G).

Summary updated

Abstract

In this paper, we propose a novel cross-modal distillation method, called TinyCLIP, for large-scale language-image pre-trained models. The method introduces two core techniques: affinity mimicking and weight inheritance. Affinity mimicking explores the interaction between modalities during distillation, enabling student models to mimic teachers' behavior of learning cross-modal feature alignment in a visual-linguistic affinity space. Weight inheritance transmits the pre-trained weights from the teacher models to their student counterparts to improve distillation efficiency. Moreover, we extend the method into a multi-stage progressive distillation to mitigate the loss of informative weights during extreme compression. Comprehensive experiments demonstrate the efficacy of TinyCLIP, showing that it can reduce the size of the pre-trained CLIP ViT-B/32 by 50%, while maintaining comparable zero-shot performance. While aiming for comparable performance, distillation with weight inheritance can speed up the training by 1.4 - 7.8 $\times$ compared to training from scratch. Moreover, our TinyCLIP ViT-8M/16, trained on YFCC-15M, achieves an impressive zero-shot top-1 accuracy of 41.1% on ImageNet, surpassing the original CLIP ViT-B/16 by 3.5% while utilizing only 8.9% parameters. Finally, we demonstrate the good transferability of TinyCLIP in various downstream tasks. Code and models will be open-sourced at https://aka.ms/tinyclip.
BibTeX
@inproceedings{wu2023tinyclip,
  author = {Kan Wu and Houwen Peng and Zhenghong Zhou and Bin Xiao and Mengchen Liu and Lu Yuan and Hong Xuan and Michael Valenzuela and Xi Chen and Xinggang Wang and Hongyang Chao and Han Hu},
  title = {{TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=8bxHTzEHIa},
}
Figure