Revisiting Prototypical Network for Cross Domain Few-Shot Learning

Fei Zhou, Peng Wang, Lei Zhang, Wei Wei, Yanning Zhang

Abstract

Prototypical Networks (ProtoNet) are a foundational method for few-shot classification but suffer significant performance degradation in cross-domain scenarios. This failure is often attributed to "simplicity bias," where models latch onto easy, non-transferable shortcuts like background color or texture. We propose the Local-global Distillation Prototypical Network (LDP-net) to address this. LDP-net utilizes a dual-branch architecture with local-global distillation and cross-episode momentum updates to force the extraction of robust, invariant semantic features. Our method achieves state-of-the-art results across eight diverse target domains, demonstrating superior generalization compared to standard ProtoNet variants.

Introduction

Few-Shot Classification (FSC) aims to recognize novel categories from extremely limited data. While Prototypical Networks are effective within a single domain, they struggle with Cross-Domain Few-Shot Learning (CD-FSL), where training and testing data come from different distributions (e.g., natural images vs. satellite imagery). The primary hurdle is simplicity bias: deep networks tend to exploit simple shortcuts present in the training domain that do not generalize. To succeed, models must capture the core semantic "essence" of objects rather than superficial background cues.

Methodology: The LDP-net Framework

LDP-net moves beyond the single-stream architecture of traditional ProtoNets by introducing a dual-branch system designed to break simplicity bias through knowledge distillation.

The LDP-net architecture featuring global and local branches with self-image and cross-image distillation mechanisms.

Dual-Branch Architecture

The framework consists of a Global Branch, which processes full query images, and a Local Branch, which processes random local crops. By comparing predictions between these branches, the model is forced to find semantic meaning even in partial views of an object.

Knowledge Distillation (KD)

The model enforces consistency between the two branches through two types of distillation:

  • Self-image Distillation: Ensures the local crop's prediction matches the global image's prediction, encouraging the network to identify semantic features within small regions.
  • Cross-image Distillation: Aligns a local crop's prediction with the global prediction of a different image from the same class. This regularizer helps the model ignore intra-class variations like pose or lighting.

Momentum Updates

To stabilize learning across diverse meta-learning episodes, the local branch is updated as an Exponential Moving Average (EMA) of the global branch. This "cross-episode distillation" allows the model to accumulate long-term knowledge, preventing overfitting to the quirks of individual tasks.

Experimental Results

LDP-net was trained on mini-ImageNet and evaluated across eight diverse target domains, ranging from fine-grained natural images (CUB, Cars) to extreme domain shifts (ChestX, EuroSAT).

Quantitative Performance

LDP-net consistently outperformed baselines. On the CUB dataset (1-shot), it achieved 49.82% accuracy, a nearly 10% improvement over ProtoNet++ (40.34%). Significant gains were also observed in satellite (EuroSAT) and agricultural (CropDisease) imagery compared to previous methods like GNN+AFA and RDC.

Qualitative Analysis

Class Activation Maps (CAM) revealed that LDP-net focuses on the entire object rather than biased corners. Furthermore, visualization of the loss landscape confirms that LDP-net produces a smoother, wider landscape, which is strongly correlated with superior generalization capability.

Comparison of loss landscapes between ProtoNet++ and LDP-net, showing the smoother and wider landscape of the proposed method.

Conclusion

LDP-net effectively mitigates simplicity bias in cross-domain few-shot learning by leveraging local-global distillation and momentum-based updates. While performance on extreme shifts like medical imaging remains a challenge, the robust semantic features learned by LDP-net represent a significant step forward for generalized few-shot classification. Future work will explore increasing training data diversity and more efficient target-domain adaptation strategies.

Summary updated

Abstract

Prototypical Network is a popular few-shot solver that aims at establishing a feature metric generalizable to novel few-shot classification (FSC) tasks using deep neural networks. However, its performance drops dramatically when generalizing to the FSC tasks in new domains. In this study, we revisit this problem and argue that the devil lies in the simplicity bias pitfall in neural networks. In specific, the network tends to focus on some biased shortcut features (e.g., color, shape, etc.) that are exclusively sufficient to distinguish very few classes in the meta-training tasks within a pre-defined domain, but fail to generalize across domains as some desirable semantic features. To mitigate this problem, we propose a Local-global Distillation Prototypical Network (LDP-net). Different from the standard Prototypical Network, we establish a two-branch network to classify the query image and its random local crops, respectively. Then, knowledge distillation is conducted among these two branches to enforce their class affiliation consistency. The rationale behind is that since such global-local semantic relationship is expected to hold regardless of data domains, the local-global distillation is beneficial to exploit some cross-domain transferable semantic features for feature metric establishment. Moreover, such local-global semantic consistency is further enforced among different images of the same class to reduce the intra-class semantic variation of the resultant feature. In addition, we propose to update the local branch as Exponential Moving Average (EMA) over training episodes, which makes it possible to better distill cross-episode knowledge and further enhance the generalization performance. Experiments on eight cross-domain FSC benchmarks empirically clarify our argument and show the state-of-the-art results of LDP-net. Code is available in https://github.com/NWPUZhoufei/LDP-Net
BibTeX
@inproceedings{zhou2023revisiting,
  author = {Fei Zhou and Peng Wang and Lei Zhang and Wei Wei and Yanning Zhang},
  title = {{Revisiting Prototypical Network for Cross Domain Few-Shot Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01921},
}
Figure