Co-Tuning for Transfer Learning

Kaichao You, Zhi Kou, Mingsheng Long, Jianmin Wang

TL;DR

Co-Tuning is a novel transfer learning framework that achieves "full transfer" by reusing the task-specific layers (the "head") of pre-trained models, which are typically discarded. By learning the probabilistic relationship between source and target categories, Co-Tuning provides an extra layer of supervision that yields up to a 20% relative performance boost over standard fine-tuning.

The Problem: Wasted Parameters

In the standard transfer learning pipeline, practitioners take a model pre-trained on a massive dataset (like ImageNet), chop off the final classification layer, and replace it with a new, randomly initialized "head" for the target task.

The authors argue this is a "reckless waste." These discarded task-specific layers can account for over 20% of a model's total parameters (e.g., in BERT-base). More importantly, these layers aren't just arbitrary weights; they contain rich categorical knowledge. However, because the categories in the source dataset (e.g., "Indian elephant") differ from the target dataset (e.g., "elephant"), practitioners haven't had a clear way to reuse them. The core challenge is translating knowledge from one set of labels to another when the label spaces are completely different.

The Co-Tuning Framework

The authors propose Co-Tuning, a two-step framework designed to exploit the entire pre-trained model:

  1. Learning Category Relationships: The core idea is to find the conditional distribution $p(y_s|y_t)$—essentially asking, "If an image belongs to target category $A$, what is the probability it belongs to source categories $X, Y, \text{ and } Z$?"
  2. Reverse Approach: First, use the pre-trained model to predict source labels for the target data. Then, learn a mapping from these source predictions back to the ground-truth target labels. Finally, use Bayes’ rule to flip this and get the relationship of source labels given target labels.
  3. Calibration: To ensure these probabilities are reliable, they use Temperature Scaling. This prevents the pre-trained model from being "over-confident" in its source predictions.
  4. Collaborative Supervision: During fine-tuning, the model is trained with two losses simultaneously. One loss comes from the standard ground-truth target labels (one-hot). The second loss comes from "probabilistic source labels" generated by translating the target label through the relationship learned in step one. This allows the pre-trained "source head" to help supervise the feature extractor alongside the new "target head."

Key Insights: Probabilistic Mapping

  • The "Aha" Moment: A target category is rarely a 1:1 match for a source category. For example, the COCO category "elephant" is actually a mixture of ImageNet's "Indian elephant," "African elephant," and "tusker." Co-Tuning captures this nuance by mapping target labels to a distribution of source labels.
Probabilistic mapping between COCO and ImageNet categories
  • Beyond Regularization: Most state-of-the-art fine-tuning methods (like L2-SP or DELTA) act as regularizers to prevent overfitting on small data. Co-Tuning is different; it provides additional information. This is why it continues to show gains on large-scale datasets where regularization-based methods typically fail to beat vanilla fine-tuning.
  • Pluggability: The method is highly flexible. Once the training is finished, the source head can be discarded, meaning there is zero extra computational cost during inference.

Experimental Results

The researchers evaluated Co-Tuning across several domains, including medium-scale vision (CUB-200-2011, Stanford Cars, FGVC Aircraft), large-scale vision (COCO-70), and NLP (CoNLL 2003 for Named Entity Recognition).

  • Significant Accuracy Gains: Co-Tuning demonstrates massive improvements in low-data regimes. On the Stanford Cars dataset with only 15% of the training data, Co-Tuning improved accuracy from 36.77% (baseline) to 46.02%—a 20% relative improvement.
Performance comparison across datasets and sampling rates
  • Consistency Across Scales: In the large-scale COCO-70 experiment, while other advanced methods performed similarly to vanilla fine-tuning, Co-Tuning maintained a clear lead, proving its robustness to dataset size.
  • Cross-Domain Success: The method successfully transferred to NLP, improving a pre-trained BERT's F1-score on the NER task from 90.81 to 91.27.

Limitations and Future Work

  • Diversity Requirement: Co-Tuning relies on the assumption that the pre-trained (source) dataset is "diverse enough" to act as a basis for the target categories. If the source and target domains are extremely disjoint (e.g., pre-training on medical X-rays to classify emojis), the categorical mapping might be less effective.
  • Scaling the Label Space: The authors note they haven't yet tested cases where the target dataset has significantly more classes than the source dataset (e.g., transferring from ImageNet-1k to a dataset with 10,000 classes). This remains an open question for future validation.

Summary updated

Abstract

Fine-tuning pre-trained deep neural networks (DNNs) to a target dataset, also known as transfer learning, is widely used in computer vision and NLP. Because task-specific layers mainly contain categorical information and categories vary with datasets, practitioners only \textit{partially} transfer pre-trained models by discarding task-specific layers and fine-tuning bottom layers. However, it is a reckless loss to simply discard task-specific parameters who take up as many as $20\%$ of the total parameters in pre-trained models. To \textit{fully} transfer pre-trained models, we propose a two-step framework named \textbf{Co-Tuning}: (i) learn the relationship between source categories and target categories from the pre-trained model and calibrated predictions; (ii) target labels (one-hot labels), as well as source labels (probabilistic labels) translated by the category relationship, collaboratively supervise the fine-tuning process. A simple instantiation of the framework shows strong empirical results in four visual classification tasks and one NLP classification task, bringing up to $20\%$ relative improvement. While state-of-the-art fine-tuning techniques mainly focus on how to impose regularization when data are not abundant, Co-Tuning works not only in medium-scale datasets (100 samples per class) but also in large-scale datasets (1000 samples per class) where regularization-based methods bring no gains over the vanilla fine-tuning. Co-Tuning relies on a typically valid assumption that the pre-trained dataset is diverse enough, implying its broad application area.
BibTeX
@inproceedings{you2020co,
  author = {Kaichao You and Zhi Kou and Mingsheng Long and Jianmin Wang},
  title = {{Co-Tuning for Transfer Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
}
Figure