Self-Attention Message Passing for Contrastive Few-Shot Learning

Ojas Kishorkumar Shirekar, Anuj Singh, Hadi Jamali Rad

Abstract

Humans have a unique ability to learn new representations from just a handful of examples with little to no supervision. Deep learning models, however, require an abundance of data and supervision to perform at a satisfactory level. Unsupervised few-shot learning (U-FSL) is the pursuit of bridging this gap between machines and humans. Inspired by the capacity of graph neural networks (GNNs) in discovering complex inter-sample relationships, we propose a novel self-attention based message passing contrastive learning approach (coined as SAMP-CLR) for U-FSL pre-training. We also propose an optimal transport (OT) based fine-tuning strategy (we call OpT-Tune) to efficiently induce task awareness into our novel end-to-end unsupervised few-shot classification framework (SAMPTransfer). Our extensive experimental results corroborate the efficacy of SAMPTransfer in a variety of downstream few-shot classification scenarios, setting a new state-of-the-art for U-FSL on both miniImagenet and tieredImagenet benchmarks, offering up to 7%+ and 5%+ improvements, respectively. Our further investigations also confirm that SAMPTransfer remains on-par with some supervised baselines on miniImagenet and outperforms all existing U-FSL baselines in a challenging cross-domain scenario.

Introduction

Deep learning models are notoriously "data-hungry," often requiring thousands of labeled examples to recognize a new category. Humans, by contrast, can learn a new concept from just one or two examples. The sub-field of Unsupervised Few-Shot Learning (U-FSL) tries to mimic this by training models on large amounts of unlabeled data so they can later adapt to new tasks with minimal supervision.

Current contrastive learning methods (like SimCLR) usually treat every single image in a batch as a unique class. This is problematic because a random batch might contain multiple images of the same semantic class; standard contrastive learning would mistakenly try to push their representations apart. SAMPTransfer aims to capture these hidden inter-sample relationships to build more robust features.

SAMP-CLR (Pre-training)

The first stage of our framework is SAMP-CLR. Instead of looking at images in isolation, the model builds a graph where each image is a node. It uses Self-Attention Message Passing (SAMP)—inspired by Transformers and GNNs—to allow nodes to "talk" to each other. This helps the model discover complex relationships between different samples in the same mini-batch.

The SAMP-CLR pre-training architecture.

The model employs a contrastive loss to ensure that an image and its augmented version have similar embeddings, while also using the SAMP-refined features to enforce broader semantic consistency across the batch.

OpT-Tune (Fine-tuning)

When presented with a new few-shot task, the model must create "prototypes" (average representations) for the new classes. We introduce OpT-Tune, which utilizes Optimal Transport (OT)—specifically the Sinkhorn-Knopp algorithm—to mathematically align the distribution of the small labeled "support set" with the unlabeled "query set."

The OpT-Tune fine-tuning framework.

This "task-awareness" makes the class prototypes much more accurate for the specific data at hand. Crucially, OpT-Tune doesn't require learning new parameters; it is a mathematical optimization that significantly improves adaptation by ensuring support examples better represent the query distribution.

Experimental Results

SAMPTransfer was evaluated on standard benchmarks including miniImageNet and tieredImageNet, as well as cross-domain scenarios (CDFSL) such as satellite imagery (EuroSAT) and medical X-rays (ChestX).

  • State-of-the-Art Performance: On miniImageNet, SAMPTransfer achieved 61.02% accuracy (1-shot), outperforming the previous best unsupervised method by over 7%.
  • Competitive with Supervised Learning: Remarkably, this unsupervised approach outperformed several well-known supervised methods (like MAML and ProtoNet) on the same benchmarks.
  • Cross-Domain Robustness: The model set new records for unsupervised learning in almost every CDFSL category.
  • Ablation Study: Removing the OpT-Tune step resulted in a 9% drop in accuracy, confirming that distribution alignment is critical for few-shot success.

Conclusion

SAMPTransfer demonstrates that GNNs are powerful tools not just for inference, but for unsupervised pre-training by helping models understand the underlying manifold of unlabeled data. While challenges remain regarding message-passing complexity and memory constraints, the combination of self-attention and optimal transport sets a new standard for unsupervised few-shot classification.

Summary updated

Abstract

Humans have a unique ability to learn new representations from just a handful of examples with little to no supervision. Deep learning models, however, require an abundance of data and supervision to perform at a satisfactory level. Unsupervised few-shot learning (U-FSL) is the pursuit of bridging this gap between machines and humans. Inspired by the capacity of graph neural networks (GNNs) in discovering complex inter-sample relationships, we propose a novel self-attention based message passing contrastive learning approach (coined as SAMP-CLR) for U-FSL pre-training. We also propose an optimal transport (OT) based fine-tuning strategy (we call OpT-Tune) to efficiently induce task awareness into our novel end-to-end unsupervised few-shot classification framework (SAMPTransfer). Our extensive experimental results corroborate the efficacy of SAMPTransfer in a variety of downstream few-shot classification scenarios, setting a new state-of-the-art for U-FSL on both miniImagenet and tieredImagenet benchmarks, offering up to 7%+ and 5%+ improvements, respectively. Our further investigations also confirm that SAMPTransfer remains on-par with some supervised baselines on miniImagenet and outperforms all existing U-FSL baselines in a challenging cross-domain scenario. Our code can be found in our GitHub repository at https://github.com/ojss/SAMPTransfer/.
BibTeX
@inproceedings{shirekar2023self,
  author = {Ojas Kishorkumar Shirekar and Anuj Singh and Hadi Jamali Rad},
  title = {{Self-Attention Message Passing for Contrastive Few-Shot Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=Fag6GvbS6au},
}
Figure