Revisiting Fine-tuning for Few-shot Learning

Akihiro Nakamura, Tatsuya HaradaarXiv 2019 · OpenReview

Abstract

Few-shot learning is the process of learning novel classes using only a few examples and it remains a challenging task in machine learning. Many sophisticated few-shot learning algorithms have been proposed based on the notion that networks can easily overfit to novel examples if they are simply fine-tuned using only a few examples. In this study, we show that in the commonly used low-resolution mini-ImageNet dataset, the fine-tuning method achieves higher accuracy than common few-shot learning algorithms in the 1-shot task and nearly the same accuracy as that of the state-of-the-art algorithm in the 5-shot task. We then evaluate our method with more practical tasks, namely the high-resolution single-domain and cross-domain tasks. With both tasks, we show that our method achieves higher accuracy than common few-shot learning algorithms. We further analyze the experimental results and show that: 1) the retraining process can be stabilized by employing a low learning rate, 2) using adaptive gradient optimizers during fine-tuning can increase test accuracy, and 3) test accuracy can be improved by updating the entire network when a large domain-shift exists between base and novel classes.

Introduction

Few-shot learning (FSL) aims to train models to recognize new classes from a minimal number of labeled examples, typically between one and five. The prevailing assumption in the field has been that standard fine-tuning is unsuitable for FSL because deep neural networks, with their millions of parameters, are prone to extreme overfitting when presented with such sparse data. This belief led to the development of complex meta-learning and metric-learning frameworks designed to bypass traditional gradient-based adaptation.

However, this research suggests that the perceived failure of fine-tuning may stem from suboptimal implementation rather than inherent limitations. By systematically re-evaluating the fine-tuning pipeline, we demonstrate that a "back-to-basics" approach can match or exceed the performance of sophisticated state-of-the-art (SOTA) algorithms.

Methodology

The proposed fine-tuning approach operates in two phases: pre-training on a large base dataset and subsequent adaptation to novel classes. To optimize this process, several architectural and procedural refinements were implemented.

Overview of the fine-tuning pipeline featuring weight imprinting and a normalized classifier to stabilize adaptation.

Key components include:

  • Weight Imprinting: Instead of random initialization, the weights for novel classes in the final layer are initialized using the average feature vectors of the provided examples. This provides a mathematically grounded starting point for optimization.
  • Normalized Classifiers: The model utilizes a cosine-similarity-based classifier, where both weights and features are normalized. This helps maintain a consistent output range and stabilizes gradients.
  • Adaptive Update Scopes: Depending on the task, the model can update only the final classification layer, the classification layer plus Batch Normalization (BN) statistics, or the entire network.

Experimental Setup

The method was evaluated across three distinct benchmarks:

  1. Low-resolution mini-ImageNet: The standard 84 × 84 pixel benchmark for FSL.
  2. High-resolution mini-ImageNet: A 224 × 224 pixel version designed to test the impact of image resolution on overfitting.
  3. Cross-domain Task: A challenging scenario where the model is pre-trained on general objects (mini-ImageNet) and tested on specific bird species (CUB-200-2011), testing robustness to significant domain shifts.

Analysis and Results

The experiments revealed that the primary barrier to successful fine-tuning in previous literature was not the parameter count, but the choice of hyperparameters—specifically the learning rate.

Comparison of training stability across different learning rates. A lower learning rate (0.0001) prevents the erratic behavior seen at higher rates.

The Importance of Stability

As shown in the analysis of learning rates, higher rates (e.g., 0.01) lead to highly erratic validation accuracy, indicative of unstable training. By reducing the learning rate to 0.0001, the retraining process becomes remarkably stable, allowing the model to adapt without catastrophic forgetting or extreme overfitting.

Performance Benchmarks

  • Standard FSL: On the 1-shot mini-ImageNet task, the fine-tuning method achieved 54.90% accuracy, surpassing complex models like MatchingNet.
  • Cross-Domain Performance: In the 5-shot cross-domain task, the method reached 74.88% accuracy, significantly outperforming RelationNet (57.71%). This success was largely attributed to updating the entire network, allowing the feature extractor to adapt to the new domain.
  • Optimizer Choice: Adaptive optimizers like Adam were found to be essential, providing a significant boost in test accuracy compared to standard Momentum-SGD.

Conclusion

This study demonstrates that simple fine-tuning is a surprisingly powerful tool for few-shot learning when properly executed. By focusing on stability through low learning rates and utilizing adaptive optimizers, fine-tuning can overcome the "overfitting myth." While updating the entire network is computationally more intensive than some metric-learning approaches, the gains in accuracy—especially in cross-domain scenarios—make it a compelling strategy for practical FSL applications.

Summary updated

Abstract

Few-shot learning is the process of learning novel classes using only a few examples and it remains a challenging task in machine learning. Many sophisticated few-shot learning algorithms have been proposed based on the notion that networks can easily overfit to novel examples if they are simply fine-tuned using only a few examples. In this study, we show that in the commonly used low-resolution mini-ImageNet dataset, the fine-tuning method achieves higher accuracy than common few-shot learning algorithms in the 1-shot task and nearly the same accuracy as that of the state-of-the-art algorithm in the 5-shot task. We then evaluate our method with more practical tasks, namely the high-resolution single-domain and cross-domain tasks. With both tasks, we show that our method achieves higher accuracy than common few-shot learning algorithms. We further analyze the experimental results and show that: 1) the retraining process can be stabilized by employing a low learning rate, 2) using adaptive gradient optimizers during fine-tuning can increase test accuracy, and 3) test accuracy can be improved by updating the entire network when a large domain-shift exists between base and novel classes.
BibTeX
@misc{nakamura2019revisiting,
  author = {Akihiro Nakamura and Tatsuya Harada},
  title = {{Revisiting Fine-tuning for Few-shot Learning}},
  year = {2019},
  eprint = {1910.00216},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.00216},
}
Figure