Abstract
We investigate cross-domain few-shot learning under the constraint that fine-tuning of backbones (i.e., feature extractors) is impossible or infeasible -- a scenario that is increasingly common in practical use cases. Handling the low-quality and static embeddings produced by frozen, "black-box" backbones leads to a problem representation of few-shot classification as a series of multiple instance verification (MIV) tasks. Inspired by this representation, we introduce a novel approach to few-shot domain adaptation, named the "MIV-head", akin to a classification head that is agnostic to any pretrained backbone and computationally efficient. The core components designed for the MIV-head, when trained on few-shot data from a target domain, collectively yield strong performance on test data from that domain. Importantly, it does so without fine-tuning the backbone, and within the "meta-testing" phase. Experimenting under various settings and on an extension of the Meta-dataset benchmark for cross-domain few-shot image classification, using representative off-the-shelf convolutional neural network and vision transformer backbones pretrained on ImageNet1K, we show that the MIV-head achieves highly competitive accuracy when compared to state-of-the-art "adapter" (or partially fine-tuning) methods applied to the same backbones, while incurring substantially lower adaptation cost. We also find well-known "classification head" approaches lag far behind in terms of accuracy. Ablation study empirically justifies the core components of our approach.
Introduction
Cross-Domain Few-Shot Learning (CDFSL) aims to adapt models trained on a source domain (e.g., ImageNet) to novel target domains (e.g., medical imaging, satellite data) using minimal labeled data. While state-of-the-art methods often utilize "adapters" — small fine-tunable modules inserted into the backbone — this approach faces significant hurdles. Modern foundation models are frequently accessed as "black boxes" via APIs, precluding weight modification. Furthermore, adapters are typically architecture-specific and computationally expensive to train due to the need for backpropagation through the entire backbone.
To address these challenges, we propose the MIV-head, a novel, backbone-agnostic classification head. By reframing few-shot classification as a series of multiple instance verification (MIV) tasks, the MIV-head achieves high accuracy with frozen backbones while significantly reducing adaptation costs.
The MIV-head Architecture
The core innovation of the MIV-head lies in treating classification as verification. Instead of a direct multi-class prediction, the model evaluates for each class: "Does this bag of support images contain the same class as this query?"
The architecture comprises three primary components:
- Pooling-by-Attention: This module replaces standard global average pooling with a learnable attention mechanism. This allows spatial patches to "compete" to represent the image, enabling the model to focus on domain-specific features even when the backbone is frozen.
- Cross-Attention Pooling (CAP): This is the heart of the verification approach. It treats each class in the support set as a "bag." The query image serves as a reference, and support images within a class compete to form a "dynamic prototype" most relevant to that specific query.
- Multi-Block Logits: Features are extracted from multiple intermediate layers (blocks) of the backbone. Similarity scores (logits) are computed independently for each block and aggregated using a logsumexp function, ensuring that the most informative layers drive the final classification decision.
Experimental Evaluation
We evaluated the MIV-head on an extended version of the Meta-Dataset (MD+), encompassing 17 diverse datasets ranging from natural images (MSCOCO) to specialized domains like skin lesions (ISIC), X-rays (ChestX), and satellite imagery (EuroSAT). Experiments were conducted across various backbones, including ResNet-50 and Vision Transformers (ViT, DeiT, Swin), using both supervised and self-supervised (DINO) pretraining.
Results and Discussion
The MIV-head demonstrates superior performance and efficiency compared to leading adapter-based methods like TSA and eTT.
Key Findings: * Accuracy: On a supervised ResNet-50 backbone, the MIV-head achieved an average accuracy of 78.3% on Meta-Dataset, significantly outperforming TSA's 74.6%. * Efficiency: Adaptation costs (measured in GFLOPs) were typically reduced by over 50% compared to leading adapter methods. Training times were also 30-50% faster, making it highly suitable for resource-constrained environments. * Robustness: The MIV-head is inherently backbone-agnostic and handles high-resolution patch settings (e.g., ViT with patch-8) without the memory overhead that often causes adapter methods to fail with "Out of Memory" errors.
The "competition" mechanism — where patches and images compete via attention — is crucial for handling static, unoptimized embeddings. By reframing the problem as verification, the model creates dynamic prototypes that adapt to the specific query, a more flexible approach than fixed class centroids.
Conclusion
The MIV-head provides a lightweight, effective path for cross-domain few-shot learning with frozen backbones. While it shows some limitations on very low-resolution data (e.g., CIFAR), its efficiency and backbone-agnostic nature make it a powerful tool for deploying large-scale foundation models in black-box scenarios. Future work will explore the integration of meta-learning to further optimize the head's initial parameters.
Code available at: https://github.com/xxweka/MIV-head
Summary updated