Abstract
Humans have a unique ability to learn new representations from just a handful of examples with little to no supervision. Deep learning models, however, require an abundance of data and supervision to perform at a satisfactory level. Unsupervised few-shot learning (U-FSL) is the pursuit of bridging this gap between machines and humans. Inspired by the capacity of graph neural networks (GNNs) in discovering complex inter-sample relationships, we propose a novel self-attention based message passing contrastive learning approach (coined as SAMP-CLR) for U-FSL pre-training. We also propose an optimal transport (OT) based fine-tuning strategy (we call OpT-Tune) to efficiently induce task awareness into our novel end-to-end unsupervised few-shot classification framework (SAMPTransfer). Our extensive experimental results corroborate the efficacy of SAMPTransfer in a variety of downstream few-shot classification scenarios, setting a new state-of-the-art for U-FSL on both miniImageNet and tieredImageNet benchmarks, offering up to 7%+ and 5%+ improvements, respectively. Our further investigations also confirm that SAMPTransfer remains on-par with some supervised baselines on miniImageNet and outperforms all existing U-FSL baselines in a challenging cross-domain scenario.
Introduction
Deep learning models typically require thousands of labeled examples to recognize new categories, whereas humans can learn from a single instance. Unsupervised Few-Shot Learning (U-FSL) addresses this gap by focusing on two main challenges: learning robust representations without labels and adapting those representations to new tasks with minimal support data. Existing methods often treat batch samples in isolation, failing to exploit the inherent structure of the data or struggling with the sample bias inherent in small support sets.
Methodology: SAMPTransfer
The proposed SAMPTransfer framework addresses these challenges through a two-phase approach: unsupervised pre-training followed by task-aware fine-tuning.
SAMP-CLR: Contextual Pre-training
Standard contrastive learning methods like SimCLR focus on instance-level discrimination, often ignoring that different images in a batch may share the same semantic class. SAMP-CLR introduces Self-Attention Message Passing (SAMP) to overcome this. By modeling a batch as a graph where images are nodes, the self-attention mechanism allows nodes to exchange information. This process refines features based on similar neighbors, effectively discovering latent clusters without explicit supervision.
OpT-Tune: Task-Aware Fine-tuning
During the few-shot classification phase, models often suffer from "sample bias" because a tiny support set may not represent the true class distribution. OpT-Tune employs Optimal Transport (OT), specifically the Sinkhorn-Knopp algorithm, to align the support set features with the distribution of the unlabeled query set. This alignment induces "task awareness," ensuring the classifier is optimized for the specific characteristics of the current task.
Experimental Evaluation
SAMPTransfer was evaluated on standard benchmarks (miniImageNet, tieredImageNet) and challenging cross-domain datasets (CUB, CDFSL benchmark).
Key Results
SAMPTransfer achieved significant performance gains over previous unsupervised state-of-the-art (SOTA) methods:
- miniImageNet: Reached 61.02% accuracy on 1-shot tasks (a ~7% improvement) and 72.52% on 5-shot tasks.
- tieredImageNet: Observed improvements of ~3% and ~5% for 1-shot and 5-shot tasks, respectively.
- Cross-Domain Robustness: In scenarios like ImageNet to X-ray (ChestX), SAMPTransfer remained competitive with or outperformed specialized cross-domain models.
Discussion and Future Work
The success of SAMPTransfer highlights the importance of context and alignment in few-shot learning. By treating batches as graphs, the model exploits intra-batch relationships to learn richer features. Furthermore, using Optimal Transport for alignment provides a parameter-free method to mitigate sample bias.
Future work includes integrating memory modules into SAMP-CLR to capture global data distributions beyond individual batches, potentially further closing the gap between unsupervised and supervised learning.
Code available at: https://github.com/ojss/SAMPTransfer/
Summary updated