TL;DR
This paper argues that simply maximizing ImageNet accuracy isn't enough for good transfer learning. Instead, they propose "Calibrated Imagenet Score" (CIS), a metric combining ImageNet accuracy with "Filter Diversity," as a better predictor of transferability. They introduce a "Controlled Label Injection" (CLI) method to create models with high CIS, demonstrating improved performance on various downstream tasks.
What Problem Does This Paper Address?
The prevailing wisdom in computer vision has been that models achieving higher accuracy on large-scale datasets like ImageNet will also perform better when fine-tuned on smaller, downstream tasks. This has led to a significant focus on optimizing ImageNet performance. However, recent work has shown that self-supervised learning (SSL) models can achieve superior transferability despite having lower ImageNet accuracy. This discrepancy highlights that ImageNet accuracy alone is not the sole determinant of a model's ability to transfer knowledge. The authors aim to identify the additional factors that contribute to good transferability, beyond just upstream task performance.
Methods
The core contribution of this paper is the introduction of Filter Diversity as a crucial factor for transferability. They propose two ways to measure this diversity:
- Clustering Filter Diversity: This method assesses how filters within a network are organized into clusters. Tightly clustered filters suggest low diversity, while sparsely spread filters indicate high diversity. They use an agglomerative clustering approach and average cluster ratios across layers.
- Spectral Filter Diversity: This approach uses Principal Component Analysis (PCA) to analyze the filter distribution. Low diversity means the distribution can be explained by a few principal directions, while high diversity requires many.
Based on these measures, they define the Calibrated Imagenet Score (CIS) as the product of ImageNet accuracy and Filter Diversity: $CIS = \text{Imagenet Accuracy} \times \text{Filter Diversity}$. They hypothesize that CIS is a better predictor of transferability than ImageNet accuracy alone.
To create models with high CIS, they introduce Controlled Label Injection (CLI). This method starts with a self-supervised pre-trained model (which typically has high filter diversity) and gradually injects label information during fine-tuning. The "control" comes from how frequently the backbone weights are updated relative to the classification head. A slower update rate for the backbone (higher control cycle $T$) allows the model to retain more of its original filter diversity while still improving ImageNet accuracy.
Datasets
The paper uses the ImageNet dataset for pre-training and evaluating ImageNet accuracy. For downstream tasks, they evaluate on:
- MS-COCO: A multi-label image classification dataset.
- 14 single-label image classification datasets, covering a range of tasks including superordinate-level object classification (CIFAR-10, CIFAR-100), fine-grained object classification (e.g., Stanford Cars, Oxford Flowers), texture classification (DTD), and scene classification (MIT indoor, SUN397).
Key Insights and Ideas
- ImageNet Accuracy is Not Enough: The paper strongly challenges the sole reliance on ImageNet accuracy for selecting pre-trained models for transfer learning.
- Filter Diversity Matters: They provide compelling evidence that the diversity of learned filters is a critical, intrinsic property that significantly boosts transferability.
- CIS as a Better Metric: The proposed Calibrated Imagenet Score (CIS) demonstrably correlates better with transferability than ImageNet accuracy alone, especially when considering both supervised and self-supervised models.
- SSL Models are Naturally Diverse: Self-supervised learning methods, by their nature of treating each instance as a unique class, tend to learn more diverse filters compared to traditional supervised methods.
- Controlled Label Injection Balances Trade-offs: The CLI method offers a principled way to inject supervised signals into SSL models, allowing for a controlled increase in ImageNet accuracy while preserving or even enhancing filter diversity, leading to improved transferability.
- Architecture Matters for Diversity: They observe that Vision Transformers (ViTs) might learn less diverse filters than CNNs when pre-trained with the same SSL method, suggesting architecture plays a role in filter diversity.
Key Results
- Improved Correlation: Both Clustering and Spectral Filter Diversity measures significantly improve the correlation between a model's score and its transferability compared to using ImageNet accuracy alone. Spectral Filter Diversity showed a slight advantage for linear probing, while Clustering Filter Diversity was better for fine-tuning.
- CLI Generates High-CIS Models: The CLI scheme successfully generates models with high Calibrated Imagenet Scores, which in turn exhibit superior transferability across various downstream tasks.
- Transferability Gains: Tables 3 and 4 show that label-injected models (using CLI) consistently outperform their base SSL counterparts and even pure supervised models on average across multiple downstream tasks, both for linear probing and fine-tuning. For instance, SwAV and DINO models, when optimized for CIS via CLI, achieve the best transferability.
- Feature Importance: XGBoost analysis confirms that CIS is the most predictive factor for transferability, significantly outperforming other metrics like CKA, intra-class variance, and class separation.
Limitations and Future Work
- Understanding SSL Diversity Mechanisms: The authors note that it's still unclear why different SSL methods result in varying levels of filter diversity, suggesting this as an area for future research.
- Diversity Regularization: They speculate that explicit diversity regularization during training might not be beneficial for complex models, as it could conflict with standard regularization techniques. However, they suggest future work could explore ways to increase "real" filter diversity.
- Scale Agnostic vs. Sensitive Measures: While Clustering Diversity is scale-agnostic, Spectral Diversity is not. The paper notes that while both are effective, their relative performance can vary depending on the specific task (linear probing vs. fine-tuning).
- ViT vs. CNN Diversity: The observation that ViTs might learn less diverse filters than CNNs warrants further investigation into the underlying reasons.
Summary updated