Self-training for Few-shot Transfer Across Extreme Task Differences

Cheng Perng Phoo, Bharath Hariharan

Abstract

Most few-shot learning techniques are pre-trained on a large, labeled "base dataset". In problem domains where such large labeled datasets are not available for pre-training (e.g., X-ray, satellite images), one must resort to pre-training in a different "source" problem domain (e.g., ImageNet), which can be very different from the desired target task. Traditional few-shot and transfer learning techniques fail in the presence of such extreme differences between the source and target tasks. In this paper, we present a simple and effective solution to tackle this extreme domain gap: self-training a source domain representation on unlabeled data from the target domain. We show that this improves one-shot performance on the target domain by 2.9 points on average on the challenging BSCD-FSL benchmark consisting of datasets from multiple domains. Our code is available at https://github.com/cpphoo/STARTUP.

Introduction

Few-shot learning (FSL) aims to train recognition systems to identify new classes using only a handful of examples. Typically, this is achieved by pre-training a model on a large, labeled "base" dataset. However, standard FSL techniques assume the base and target domains are similar—for instance, training on internet images of common animals to classify novel bird species.

This assumption breaks down in specialized domains like satellite imagery or medical diagnostics, where massive labeled base datasets do not exist. Researchers are often forced to use ImageNet as a source, creating an extreme domain gap. In these scenarios, state-of-the-art FSL methods frequently perform worse than a naive transfer learning baseline (a linear classifier on top of a frozen ImageNet CNN). Conversely, pure self-supervised learning on the target domain's unlabeled data ignores the valuable inductive biases learned from the source. This paper introduces a method to leverage both source knowledge and target-specific unlabeled data.

The STARTUP Method

We propose STARTUP (Self Training to Adapt Representations To Unseen Problems). The framework bridges the domain gap by adapting feature representations to a novel domain with a disjoint label space using a three-step process:

  1. Teacher Training: A standard classification model (the teacher) is trained on the labeled base dataset (e.g., ImageNet).
  2. Soft Label Generation: The teacher model evaluates unlabeled data from the target domain, outputting probability distributions (soft pseudo-labels) for each image.
  3. Student Training: A new student model is trained to minimize a composite loss function consisting of standard cross-entropy on the labeled base dataset, KL divergence loss to mimic the teacher's soft labels on the unlabeled target data, and a self-supervised contrastive loss (SimCLR) on the target data to learn domain-specific features.
Overview of the STARTUP framework, illustrating the representation learning phase using labeled base and unlabeled target data, followed by evaluation on few-shot tasks.

Experimental Evaluation

We evaluated STARTUP using the BSCD-FSL (Broader Study of Cross-Domain Few-Shot Learning) benchmark, which includes four datasets with extreme domain shifts from standard internet images: CropDiseases (plant leaves), EuroSAT (satellite imagery), ISIC2018 (skin lesions), and ChestX (chest X-rays). Base datasets included miniImageNet, tieredImageNet, and the full ImageNet (ILSVRC 2012).

Key Results

  • Performance Gains: STARTUP improved 1-shot classification accuracy by an average of 2.9 percentage points over the strongest baseline (naive Transfer) across the BSCD-FSL benchmark. Specific gains were even more pronounced; for example, the CropDisease dataset saw a jump of nearly 6 points (from 69.97% to 75.93%).
  • Outperforming SOTA: STARTUP consistently outperformed popular meta-learning methods like MAML, ProtoNets, and MetaOptNet, which often fail to beat naive transfer in extreme cross-domain settings.
  • Synergy of Knowledge: The method also outperformed baselines relying purely on self-supervised learning (SimCLR), proving that retaining source domain inductive biases via the teacher model is critical.

Insights and Discussion

Visual analysis confirms that STARTUP successfully adapts representations. Even though an ImageNet-trained teacher has never seen an X-ray, its predictions still group target images in structurally meaningful ways. STARTUP forces the student model to emphasize these teacher-induced groupings while refining them with self-supervised features.

t-SNE visualizations for EuroSAT and CropDisease datasets showing the feature space before and after applying STARTUP. The "After" plots show significantly more distinct and cohesive class clusters.

Our ablations reveal that target-specific unlabeled data is mandatory; using mismatched unlabeled data (e.g., using EuroSAT data to help classify CropDisease) actually degrades performance. Furthermore, while larger base models provide a better starting point, STARTUP consistently improves performance regardless of the teacher's initial size.

Conclusion

STARTUP provides a simple yet effective strategy for few-shot learning across extreme domain gaps. By combining self-training with self-supervised learning, it adapts source representations to target domains where labeled data is scarce but unlabeled data is available. Future work may explore swapping in more advanced self-supervised loss functions to further enhance cross-domain adaptation.

Summary updated

Abstract

Most few-shot learning techniques are pre-trained on a large, labeled "base dataset". In problem domains where such large labeled datasets are not available for pre-training (e.g., X-ray, satellite images), one must resort to pre-training in a different "source" problem domain (e.g., ImageNet), which can be very different from the desired target task. Traditional few-shot and transfer learning techniques fail in the presence of such extreme differences between the source and target tasks. In this paper, we present a simple and effective solution to tackle this extreme domain gap: self-training a source domain representation on unlabeled data from the target domain. We show that this improves one-shot performance on the target domain by 2.9 points on average on the challenging BSCD-FSL benchmark consisting of datasets from multiple domains. Our code is available at https://github.com/cpphoo/STARTUP.
BibTeX
@inproceedings{phoo2021self,
  author = {Cheng Perng Phoo and Bharath Hariharan},
  title = {{Self-training for Few-shot Transfer Across Extreme Task Differences}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=FthL5PzQdCB},
}
Figure