Revisit Finetuning strategy for Few-Shot Learning to Transfer the Emdeddings

Heng Wang, Tan Yue, Xiang Ye, Zihang He, Bohan Li, Yong LiICLR 2023

This paper tackles a fundamental tension in Few-Shot Learning (FSL): how to adapt a powerful pre-trained model to new classes without "breaking" it through overfitting. The authors introduce a two-stage strategy called LP-FT-FB that uses a clever "inverse" bias reduction technique to ensure that feature extraction remains accurate even when data is scarce.

By combining unbiased linear probing with a novel inverse Firth Bias Reduction (i-FBR), the proposed method prevents over-parameterized models from overfitting, achieving state-of-the-art results on standard benchmarks and cross-domain tasks.

The Problem: Feature Distortion vs. Overfitting

In Few-Shot Learning, the goal is to teach a model to recognize new categories using only a handful of examples (e.g., 1 or 5 images). Traditionally, there are two ways to approach this:

  1. Linear Probing (LP): Freeze the pre-trained feature extractor and only train a new "head" (classifier). This is stable but assumes the extractor is already perfect for the new data. This is often untrue, especially if the new data is Out-of-Distribution (OOD).
  2. Finetuning (FT): Update the entire model. While this allows the extractor to adapt, it almost always leads to massive overfitting because the model has millions of parameters and only a few images to learn from.

The authors identify a specific issue known as feature distortion. When a pre-trained extractor isn't adapted to novel samples, it projects them into a feature space designed for old classes, leading to inaccurate representations. However, standard finetuning to fix this distortion usually results in the model simply memorizing the few available samples.

Methods: The LP-FT-FB Pipeline

To resolve this tension, the authors propose a two-stage pipeline: Linear-Probing-Finetuning with Firth-Bias (LP-FT-FB).

Overview of the LP-FT-FB pipeline
  • Stage 1: Linear Probing with FBR. First, they train a linear classifier while keeping the feature extractor frozen. They utilize Firth Bias Reduction (FBR), a statistical method to obtain an unbiased estimate of the classifier's weights. FBR encourages the model to be more confident, pushing outputs away from a uniform distribution.
  • Stage 2: Finetuning with i-FBR. This is the core innovation. The authors finetune the entire model but introduce inverse Firth Bias Reduction (i-FBR). While standard FBR helps simple linear models, the authors found it actually hurts complex, over-parameterized feature extractors. By "inverting" the logic (using a negative scaling factor), i-FBR encourages the output distribution to be closer to uniform. This acts as a powerful regularizer—similar to label smoothing—that prevents the feature extractor from overfitting to the tiny novel dataset.

Key Insights

  • The Distortion Discovery: The paper theoretically and experimentally shows that if you don't update the feature extractor, the features for novel classes remain "distorted" because the model tries to force-fit them into the subspace of the base (training) classes.
  • The "Inverse" Logic: The most surprising insight is that bias reduction isn't one-size-fits-all. A technique that helps a small model (the classifier head) reach the "truth" faster actually causes a large model (the backbone) to overfit. Flipping the sign of the penalty (i-FBR) provides the necessary "drag" to keep the backbone from memorizing noise.
  • Gradient Control: i-FBR effectively decreases the gradient for the correct class and increases it for incorrect classes during training. This prevents the model from becoming "too certain" too quickly.

Key Results

The method was evaluated on the "Big Three" of FSL benchmarks: mini-ImageNet (100 classes), tiered-ImageNet (608 classes), and CUB (fine-grained bird species). They also tested cross-domain transferability by training on ImageNet and testing on CUB.

Performance comparison on standard FSL benchmarks

The proposed LP-FT-FB consistently outperformed previous State-of-the-Art (SOTA) methods across different architectures:

  • mini-ImageNet (5-way 1-shot): Achieved 67.04% accuracy, outperforming the previous best FBR method (65.59%) and S2M2 (64.48%).
  • tiered-ImageNet (5-way 5-shot): Reached 92.06%, a significant jump over standard baselines.
  • Cross-Domain Performance: In the difficult mini-ImageNet $\rightarrow$ CUB task, the method outperformed existing distribution calibration (DC) techniques by over 2% in several categories, proving that the adapted features are more robust.

Limitations and Future Work

While highly effective, the approach has a few limitations:

  • Computational Cost: Because the method involves finetuning the entire backbone (a "unit-wise" approach), it is more computationally expensive than simple linear probing. The authors liken the cost to first-order gradient methods like Reptile.
  • Granularity of Finetuning: Currently, the method treats the whole backbone as one unit. Future work could explore filter-wise or layer-wise finetuning to see if specific parts of the network should be regularized differently.
  • Hyperparameter Sensitivity: The i-FBR factor ($\lambda_{inv}$) requires tuning. The authors found $-1e^{-3}$ worked best for their setup, but this optimal value may vary across different architectures.

Summary updated

BibTeX
@inproceedings{wang2023revisit,
  author = {Heng Wang and Tan Yue and Xiang Ye and Zihang He and Bohan Li and Yong Li},
  title = {{Revisit Finetuning strategy for Few-Shot Learning to Transfer the Emdeddings}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
}
Figure