Abstract
Few-shot learning (FSL) aims to classify novel categories using only a handful of labeled examples. Traditional methods relying on cross-entropy loss often struggle with insufficient data, leading to poor generalization. This paper proposes SPTA (Supervised Pretraining with TIM Adaptation), a framework that leverages Supervised Contrastive Learning (SupCon) to enhance feature extractor quality. By creating more robust, cluster-aware embeddings during pretraining, SPTA achieves state-of-the-art performance on Mini-ImageNet and CUB benchmarks. Furthermore, the method effectively mitigates domain-shift degradation and enables a 3.87× runtime speedup in resource-constrained environments through optimized inference.
Introduction
Deep learning's success is largely predicated on the availability of massive labeled datasets. However, in many real-world scenarios, obtaining such data is impractical. Few-shot learning addresses this by training models to generalize from very few samples. The standard FSL pipeline involves pretraining a feature extractor on a large base dataset and then adapting it to novel classes.
The authors identify that the primary bottleneck in this pipeline is the feature extractor's quality. Standard Cross-Entropy (CE) loss, while effective for classification, often fails to produce features that are sufficiently discriminative for unseen classes, particularly when the base dataset is small or when a domain shift exists between base and novel sets.
The SPTA Framework
The proposed SPTA framework introduces a two-stage pretraining strategy designed to maximize the utility of the base dataset.
Supervised Contrastive Pretraining
Instead of relying solely on label prediction, SPTA utilizes Supervised Contrastive Learning. In this stage, the model is trained to minimize the distance between embeddings of the same class while maximizing the distance between different classes. This process, involving a projection head and a contrastive loss, results in a feature space where classes are represented as tight, well-separated clusters.
Two-Stage Learning and Adaptation
Following the initial contrastive phase, the projection head is discarded, and the feature extractor is fine-tuned using standard cross-entropy loss. For the final few-shot task, the framework employs Transductive Information Maximization (TIM). Unlike inductive methods that process test images individually, TIM considers the entire query set simultaneously, optimizing for confident and diverse predictions across the novel classes.
Experimental Evaluation
The researchers evaluated SPTA across three major benchmarks:
- Mini-ImageNet: A standard 100-class subset of ImageNet.
- Tiered-ImageNet: A larger, hierarchical dataset with 608 classes.
- CUB (Caltech-UCSD Birds-200-2011): A fine-grained dataset focused on bird species.
The results demonstrate that SPTA consistently outperforms existing methods. On Mini-ImageNet (5-way 1-shot), the model achieved 78.83% accuracy using a ResNet-18 backbone. On the CUB dataset, it reached an impressive 88.81% for 1-shot and 93.11% for 5-shot classification.
Efficiency and Domain Shift
A key finding of the study is the "data multiplier" effect of SupCon. The method allows models trained on smaller datasets (like CUB) to match or exceed the performance of models trained on much larger datasets (like Tiered-ImageNet) using standard techniques. This efficiency is crucial for mitigating domain shift, where performance typically drops without massive data increases.
Furthermore, the high quality of the learned features allows for faster convergence during the adaptation phase. By implementing early stopping in the TIM adaptation process on a MobileNet backbone, the authors achieved a 3.87× reduction in runtime on NVIDIA Jetson TX2 hardware, making it highly suitable for edge computing applications.
Conclusion
SPTA demonstrates that the key to superior few-shot learning lies in the quality of the initial feature representation. By replacing or augmenting standard pretraining with supervised contrastive learning, models can become significantly more data-efficient and robust. Future work may focus on further reducing the computational cost of the contrastive pretraining phase and exploring its applicability to extreme domain-shift scenarios.
Summary updated