TL;DR
Researchers at Cornell have developed STARTUP, a simple yet powerful method to improve few-shot learning when the source and target domains are radically different (e.g., internet photos vs. X-rays). By using a source-trained teacher to "group" unlabeled target data, they achieve a 2.9-point average performance boost over standard transfer learning on challenging cross-domain benchmarks.
What Problem Does This Paper Address?
Few-shot learning (FSL) aims to train models that can recognize new classes from just a handful of examples. Usually, these models are pre-trained on a massive "base" dataset (like ImageNet) and then "meta-learned" to adapt quickly to new tasks.
However, there is a catch: most FSL algorithms assume the base dataset and the new tasks come from the same domain. When the gap is "extreme"—such as transferring knowledge from pictures of cats and dogs to identifying pneumonia in X-rays or crop types in satellite imagery—traditional FSL methods fall apart. In fact, recent studies show that a "naive" transfer (just freezing a pre-trained model and training a linear layer) often beats sophisticated meta-learning algorithms in these scenarios.
The authors tackle this conundrum by leveraging the one thing many of these specialized domains have in abundance: unlabeled data.
Methods
The authors introduce STARTUP (Self-Training to Adapt Representations To Unseen Problems). The approach follows a three-step pipeline during the representation learning phase to bridge the gap between source and target domains.
- Teacher Training: Train a standard classification model (the teacher) on the labeled source "base" dataset (e.g., miniImageNet).
- Soft Labeling: Run the unlabeled target domain data through this teacher. Even though the teacher’s classes (like "husky" or "piano") are irrelevant to the target domain (like "melanoma"), the teacher’s predictions create a "soft labeling" that induces a grouping of similar images in the target domain.
- Student Training: Train a new student model by optimizing a combined loss function:
- Supervised Loss: Standard cross-entropy on the labeled source data.
- Self-Training Loss: A KL-divergence loss that forces the student to mimic the teacher’s "groupings" on the unlabeled target data.
- Self-Supervised Loss: An auxiliary SimCLR loss (contrastive learning) on the target data to help the model learn domain-specific features without any labels.
Finally, for evaluation, the representation is frozen, and a simple linear classifier is trained on the few labeled examples provided for the new task.
Datasets
The primary evaluation uses the BSCD-FSL (Broader Study of Cross-Domain Few-Shot Learning) benchmark. This benchmark uses miniImageNet as the source and tests on four very different target domains:
- ChestX: Chest X-ray images.
- ISIC2018: Skin lesion dermoscopy images.
- EuroSAT: Satellite images for land-use classification.
- CropDiseases: Images of plant leaves.
The authors also test on more standard FSL datasets like tieredImageNet to see if the method holds up when the domain gap is smaller.
Key Insights and Ideas
- Teacher-Induced Groupings: The most surprising insight is that a teacher trained on internet photos provides a useful "prior" for vastly different domains. Even if the teacher is "wrong" about the labels, its ability to say "these two images look like the same kind of thing" helps the student organize the feature space effectively. The visualizations above demonstrate how STARTUP dramatically improves the clustering of target domain data compared to baseline methods.
- Beyond Noise: The authors tested whether the unlabeled data was just providing "noise" that made the model more robust (a common theory in self-training). They found this was false; using unlabeled data from a different domain than the target actually hurt performance, proving that STARTUP is learning inherent structure specific to the target domain.
- The Sweet Spot: Self-supervised learning (like SimCLR) usually requires millions of images to work from scratch. STARTUP finds a "sweet spot" by using the source-domain knowledge to guide the self-supervised process, making it much more data-efficient.
Key Results
- State-of-the-Art Performance: STARTUP outperformed all standard few-shot learners and naive transfer learning. It showed an average improvement of 2.9 points on 1-shot tasks across the BSCD-FSL benchmark.
- Massive Gains on Specific Tasks: On the CropDiseases dataset, STARTUP provided a nearly 6-point increase for 1-shot classification compared to the previous best-performing baseline.
- Robustness to Teacher Quality: Using a stronger teacher (trained on the full ImageNet-21k) further improved results, suggesting the method scales well with better base models.
Limitations and Future Work
- Dependency on Unlabeled Data: STARTUP requires a pool of unlabeled data from the target domain during the pre-training phase. While often available, this isn't always a given in every niche application.
- The "Hardest" Gaps: While STARTUP improved performance on ChestX (the most difficult transfer in the benchmark), the gains there were smaller than in other domains, suggesting that extremely low-contrast or specialized medical imaging still poses a significant challenge.
- Inference Agnostic: The authors note that while they used a simple linear classifier for the final step, STARTUP could potentially be combined with more complex inference-time FSL algorithms (like ProtoNets) for even better results.
Summary updated