Self-Attention Message Passing for Contrastive Few-Shot Learning

Ojas Kishorkumar Shirekar, Anuj Singh, Hadi Jamali Rad

Abstract

Humans have a unique ability to learn new representations from just a handful of examples with little to no supervision. Deep learning models, however, require an abundance of data and supervision to perform at a satisfactory level. Unsupervised few-shot learning (U-FSL) is the pursuit of bridging this gap between machines and humans. Inspired by the capacity of graph neural networks (GNNs) in discovering complex inter-sample relationships, we propose a novel self-attention based message passing contrastive learning approach (coined as SAMP-CLR) for U-FSL pre-training. We also propose an optimal transport (OT) based fine-tuning strategy (we call OpT-Tune) to efficiently induce task awareness into our novel end-to-end unsupervised few-shot classification framework (SAMPTransfer). Our extensive experimental results corroborate the efficacy of SAMPTransfer in a variety of downstream few-shot classification scenarios, setting a new state-of-the-art for U-FSL on both miniImageNet and tieredImageNet benchmarks, offering up to 7%+ and 5%+ improvements, respectively. Our further investigations also confirm that SAMPTransfer remains on-par with some supervised baselines on miniImageNet and outperforms all existing U-FSL baselines in a challenging cross-domain scenario.

Introduction

Deep learning models typically require thousands of labeled examples to recognize new categories, whereas humans can learn from a single instance. Unsupervised Few-Shot Learning (U-FSL) addresses this gap by focusing on two main challenges: learning robust representations without labels and adapting those representations to new tasks with minimal support data. Existing methods often treat batch samples in isolation, failing to exploit the inherent structure of the data or struggling with the sample bias inherent in small support sets.

Methodology: SAMPTransfer

The proposed SAMPTransfer framework addresses these challenges through a two-phase approach: unsupervised pre-training followed by task-aware fine-tuning.

SAMP-CLR: Contextual Pre-training

Standard contrastive learning methods like SimCLR focus on instance-level discrimination, often ignoring that different images in a batch may share the same semantic class. SAMP-CLR introduces Self-Attention Message Passing (SAMP) to overcome this. By modeling a batch as a graph where images are nodes, the self-attention mechanism allows nodes to exchange information. This process refines features based on similar neighbors, effectively discovering latent clusters without explicit supervision.

The SAMP-CLR architecture utilizes self-attention message passing to refine image representations within a batch.

OpT-Tune: Task-Aware Fine-tuning

During the few-shot classification phase, models often suffer from "sample bias" because a tiny support set may not represent the true class distribution. OpT-Tune employs Optimal Transport (OT), specifically the Sinkhorn-Knopp algorithm, to align the support set features with the distribution of the unlabeled query set. This alignment induces "task awareness," ensuring the classifier is optimized for the specific characteristics of the current task.

OpT-Tune uses Optimal Transport to align support features with the query distribution for better task adaptation.

Experimental Evaluation

SAMPTransfer was evaluated on standard benchmarks (miniImageNet, tieredImageNet) and challenging cross-domain datasets (CUB, CDFSL benchmark).

Key Results

SAMPTransfer achieved significant performance gains over previous unsupervised state-of-the-art (SOTA) methods:

  • miniImageNet: Reached 61.02% accuracy on 1-shot tasks (a ~7% improvement) and 72.52% on 5-shot tasks.
  • tieredImageNet: Observed improvements of ~3% and ~5% for 1-shot and 5-shot tasks, respectively.
  • Cross-Domain Robustness: In scenarios like ImageNet to X-ray (ChestX), SAMPTransfer remained competitive with or outperformed specialized cross-domain models.

Discussion and Future Work

The success of SAMPTransfer highlights the importance of context and alignment in few-shot learning. By treating batches as graphs, the model exploits intra-batch relationships to learn richer features. Furthermore, using Optimal Transport for alignment provides a parameter-free method to mitigate sample bias.

Future work includes integrating memory modules into SAMP-CLR to capture global data distributions beyond individual batches, potentially further closing the gap between unsupervised and supervised learning.

Code available at: https://github.com/ojss/SAMPTransfer/

Summary updated

Abstract

Humans have a unique ability to learn new representations from just a handful of examples with little to no supervision. Deep learning models, however, require an abundance of data and supervision to perform at a satisfactory level. Unsupervised few-shot learning (U-FSL) is the pursuit of bridging this gap between machines and humans. Inspired by the capacity of graph neural networks (GNNs) in discovering complex inter-sample relationships, we propose a novel self-attention based message passing contrastive learning approach (coined as SAMP-CLR) for U-FSL pre-training. We also propose an optimal transport (OT) based fine-tuning strategy (we call OpT-Tune) to efficiently induce task awareness into our novel end-to-end unsupervised few-shot classification framework (SAMPTransfer). Our extensive experimental results corroborate the efficacy of SAMPTransferin a variety of downstream few-shot classification scenarios, setting a new state-of-the-art for U-FSL on both miniImageNet and tieredImageNet benchmarks, offering up to 7%+ and 5%+ improvements, respectively. Our further investigations also confirm that SAMPTransferremains on-par with some supervised baselines on miniImageNet and outperforms all existing U-FSL baselines in a challenging cross-domain scenario. Our code can be found in our GitHub repository: https://github.com/ojss/SAMPTransfer/.
BibTeX
@inproceedings{shirekar2023self,
  author = {Ojas Kishorkumar Shirekar and Anuj Singh and Hadi Jamali Rad},
  title = {{Self-Attention Message Passing for Contrastive Few-Shot Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=Fag6GvbS6au},
}
Figure