Shift and Scale is Detrimental To Few-Shot Transfer

Moslem Yazdanpanah, Aamer Abdul Rahman, Christian Desrosiers, Mohammad Havaei, Eugene Belilovsky, Samira Ebrahimi KahouDistShift @ NeurIPS 2021 · OpenReview

TL;DR

This paper investigates the role of the "shift and scale" (affine) parameters in Batch Normalization (BN) for few-shot learning (FSL). They find that disabling these parameters, effectively using "Feature Normalization" (FN), surprisingly improves FSL performance, especially when there's a significant domain shift between training and testing data. This simple modification offers competitive results compared to more complex FSL techniques.

What Problem Does This Paper Address?

Few-shot learning (FSL) aims to train models that can quickly adapt to new classes with very few labeled examples. A common component in deep learning models used for FSL is Batch Normalization (BN). BN normalizes layer inputs to stabilize training and speed up convergence. Crucially, BN layers typically include trainable "affine parameters" (a learned shift and scale) that adjust the normalized features. While these parameters are known to help convergence on the source task, their impact on downstream transfer performance in FSL, particularly when the target domain differs significantly from the source, is less understood. This paper questions whether these learned affine parameters are always beneficial for FSL, especially in challenging cross-domain scenarios.

Methods

The core of this paper's contribution lies in a simple yet effective modification to the standard Batch Normalization layer. They propose using "Feature Normalization" (FN), which is essentially BN without the learned affine parameters (shift and scale).

Here's a breakdown:

  • Standard Batch Normalization (BN):
    1. Normalization: Input features are normalized by subtracting the batch mean and dividing by the batch standard deviation across spatial dimensions for each channel.
  • Affine Transformation: The normalized features are then scaled by a learned parameter $\gamma$ (gamma) and shifted by a learned parameter $\beta$ (beta). These are the "shift and scale" parameters.
  • Feature Normalization (FN) (Proposed):
    1. Normalization: Same as step 1 in BN.
  • No Affine Transformation: The normalized features are used directly without any learned scaling or shifting.

The authors implement FN by simply disabling the affine parameters in standard BN modules. They then evaluate this FN approach on existing FSL methods and benchmarks.

Datasets

The paper uses the following datasets:

  • miniImageNet: Used as the base dataset for representation learning in the standard transfer learning baseline.
  • Cross-Domain Few-Shot Learning (CD-FSL) Benchmark: This benchmark comprises four target datasets with significant domain shifts:
  • EuroSAT: Satellite imagery for land usage.
  • CropDiseases: Plant images for disease identification.
  • ChestX: Chest X-rays for diagnosis.
  • ISIC2018: Skin images for melanoma detection.
  • ImageNet (Unseen Classes): Used for evaluating the baseline models in a few-shot setting with classes not seen during training.

Key Insights and Ideas

  • Affine Parameters are Detrimental to FSL Transfer: The most surprising finding is that removing the learned shift and scale parameters from Batch Normalization improves performance on few-shot transfer tasks, especially in cross-domain settings. This suggests that these parameters, while useful for source task convergence, might overfit to the source domain's statistics and hinder adaptation to novel target domains.
  • Feature Normalization is Sufficient: The paper demonstrates that the normalization step alone (Feature Normalization) is sufficient to achieve strong few-shot generalization. The learned affine transformations, which add complexity and parameters, are not necessary and can even be harmful for transfer.
  • Simplicity Wins: The proposed FN modification is extremely simple to implement (just disabling parameters) and yet achieves results comparable to, or better than, more complex and computationally expensive state-of-the-art FSL methods like STARTUP.
  • Domain Shift Matters: The performance gains from using FN are more pronounced when the domain shift between the source and target datasets is larger. This reinforces the idea that affine parameters might be more sensitive to domain-specific statistics.

Key Results

  • Improved FSL Accuracy: On the CD-FSL benchmark, models configured with Feature Normalization (FN) consistently outperformed their Batch Normalization (BN) counterparts across all datasets and shot levels (1, 5, 20, 50). For instance, on the EuroSAT dataset in the 1-shot setting, FN improved the baseline accuracy by nearly 4 percentage points.
  • Competitive with SOTA: The simple FN-configured baseline achieved results rivaling the complex STARTUP method, which relies on extensive self-training and self-supervised learning.
  • No Significant Loss on In-Distribution Tasks: While FN excels in cross-domain FSL, the paper notes that on tasks with smaller domain shifts (like the ImageNet unseen classes experiment), FN does not provide substantial gains over BN. In fact, the affine parameters might still aid convergence on in-distribution validation data, as suggested by Figure 2.

Limitations and Future Work

  • Focus on Specific Architectures: The experiments were primarily conducted using the ResNet-10 architecture. The authors suggest exploring the effect of FN on other architectures.
  • Limited Scope of Affine Parameter Study: While they show FN is beneficial, a deeper theoretical understanding of why the affine parameters are detrimental in FSL transfer, especially concerning distributional shifts, could be further investigated.
  • Future Experiments: The authors explicitly state they will consider further experiments on the effect of affine parameters in other few-shot benchmarks and methods. They also mention that this is an initial study.

Summary updated

Abstract

Batch normalization is a common component in computer vision models, including ones typically used for few-shot learning. Batch normalization applied in convolutional networks consists of a normalization step, followed by the application of per-channel trainable affine parameters which shift and scale the normalized features. The use of these affine parameters can speed up model convergence on a source task. However, we demonstrate in this work that, on common few-shot learning benchmarks, training a model on a source task using these affine parameters is detrimental to downstream transfer performance. We study this effect for several methods on well-known benchmarks such as cross-domain few-shot learning (CD-FSL) benchmark and few-shot image classification on miniImageNet. We find consistent performance gains, particularly in settings with more distant transfer tasks. Improvements from applying this low-cost and easy-to-implement modifications are shown to rival gains obtained by more sophisticated and costly methods.
BibTeX
@inproceedings{yazdanpanah2021shift,
  author = {Moslem Yazdanpanah and Aamer Abdul Rahman and Christian Desrosiers and Mohammad Havaei and Eugene Belilovsky and Samira Ebrahimi Kahou},
  title = {{Shift and Scale is Detrimental To Few-Shot Transfer}},
  booktitle = {DistShift @ NeurIPS},
  year = {2021},
  url = {https://openreview.net/forum?id=7OmUSzlgd4a},
}
Figure