Abstract
This paper addresses the few-shot image classification problem, where the classification task is performed on unlabeled query samples given a small amount of labeled support samples only. One major challenge of the few-shot learning problem is the large variety of object visual appearances that prevents the support samples to represent that object comprehensively. This might result in a significant difference between support and query samples, therefore undermining the performance of few-shot algorithms. In this paper, we tackle the problem by proposing Few-shot Cosine Transformer (FS-CT), where the relational map between supports and queries is effectively obtained for the few-shot tasks. The FS-CT consists of two parts, a learnable prototypical embedding network to obtain categorical representations from support samples with hard cases, and a transformer encoder to effectively achieve the relational map from two different support and query samples. We introduce Cosine Attention, a more robust and stable attention module that enhances the transformer module significantly and therefore improves FS-CT performance from 5% to over 20% in accuracy compared to the default scaled dot-product mechanism. Our method performs competitive results in mini-ImageNet, CUB-200, and CIFAR-FS on 1-shot learning and 5-shot learning tasks across backbones and few-shot configurations. We also developed a custom few-shot dataset for Yoga pose recognition to demonstrate the potential of our algorithm for practical application. Our FS-CT with cosine attention is a lightweight, simple few-shot algorithm that can be applied for a wide range of applications, such as healthcare, medical, and security surveillance. The official implementation code of our Few-shot Cosine Transformer is available at https://github.com/vinuni-vishc/Few-Shot-Cosine-Transformer
Introduction
Few-shot learning (FSL) aims to classify new images (queries) using only a handful of labeled examples (supports). This is notoriously difficult because a single "support" image might be a "hard case"—an odd angle or lighting—that doesn't capture the essence of the category. Furthermore, there is often a significant visual gap between the few support samples and the unlabeled query samples.
Standard Transformer models use "scaled dot-product attention," which is sensitive to feature magnitude. In few-shot scenarios, these magnitudes can vary wildly, leading to unstable "relational maps" between supports and queries. The FS-CT framework addresses these issues by refining how categorical prototypes are formed and how attention is computed between supports and queries.
Methodology: Few-shot Cosine Transformer (FS-CT)
The authors propose the Few-shot Cosine Transformer (FS-CT), a transductive learning framework that introduces two primary innovations:
Learnable Prototypical Representation
Traditional methods average support features to create a category "prototype." FS-CT uses a learnable weighted mean. This allows the model to "de-prioritize" outlier support samples and create more robust categorical representations, effectively handling "hard cases" in the support set.
Cosine Attention
Instead of the standard dot-product ($q \cdot k$), FS-CT uses cosine similarity to calculate attention weights. Because cosine similarity is bounded between -1 and 1, it ignores feature magnitude (which can be noisy) and focuses entirely on the "content" or angle of the features. This results in a much more stable and accurate correlation map between support prototypes and query images.
Cosine Linear Layer
For the final classification step, a cosine-based linear layer is used to maintain consistency in how similarity is measured throughout the network, ensuring that the entire pipeline is invariant to feature magnitude noise.
Experimental Evaluation
The model was evaluated on three gold-standard few-shot benchmarks: mini-ImageNet, CUB-200, and CIFAR-FS. Additionally, the authors introduced a custom Yoga Pose Recognition dataset containing 50 categories and ~2,500 images, serving as a real-world test for viewpoint and body type variations.
Key Results
Replacing standard attention with Cosine Attention improved FS-CT performance by 5% to 20% depending on the task and backbone. On 5-way 5-shot tasks using a ResNet-34 backbone, FS-CT achieved state-of-the-art results:
- 91.33% on mini-ImageNet.
- 91.96% on CUB-200.
- 81.46% on CIFAR-FS.
Discussion and Conclusion
In low-data regimes, the "length" (magnitude) of a feature vector often carries less useful information than its "direction." By using Cosine Attention, the model becomes invariant to these noisy magnitude fluctuations. Heatmap analysis demonstrates that Cosine Attention produces a clear, diagonal-dominant similarity matrix much earlier in the training process than standard attention, indicating faster and more stable convergence.
While performance remains tied to the choice of the embedding backbone, FS-CT is a lightweight and effective solution for practical applications in healthcare, medical imaging, and security surveillance. Future work may explore further variations of the attention mechanism and improved methods for balancing support samples during prototype generation.
Summary updated