Transfer of Pretrained Model Weights Substantially Improves Semi-Supervised Image Classification

Attaullah Sahito, Eibe Frank, Bernhard Pfahringer

Abstract

Deep neural networks produce state-of-the-art results when trained on a large number of labeled examples but tend to overfit when small amounts of labeled examples are used for training. Creating a large number of labeled examples requires considerable resources, time, and effort. If labeling new data is not feasible, so-called semi-supervised learning can achieve better generalisation than purely supervised learning by employing unlabeled instances as well as labeled ones. The work presented in this paper is motivated by the observation that transfer learning provides the opportunity to potentially further improve performance by exploiting models pretrained on a similar domain. More specifically, we explore the use of transfer learning when performing semi-supervised learning using self-learning. The main contribution is an empirical evaluation of transfer learning using different combinations of similarity metric learning methods and label propagation algorithms in semi-supervised learning. We find that transfer learning always substantially improves the model's accuracy when few labeled examples are available, regardless of the type of loss used for training the neural network. This finding is obtained by performing extensive experiments on the SVHN, CIFAR10, and Plant Village image classification datasets and applying pretrained weights from Imagenet for transfer learning.

Introduction

Training deep learning models for image classification typically requires large, fully labeled datasets. When labeling budgets are tight, models tend to overfit. Semi-supervised learning (SSL) tackles this by leveraging a small amount of labeled data alongside a large pool of unlabeled data. However, many SSL methods, such as pseudo-labeling, traditionally start training from scratch with randomly initialized weights. This paper investigates the impact of initializing SSL pipelines with weights pretrained on a massive dataset like ImageNet, exploring how transfer learning interacts with various loss functions in a self-learning framework.

Methodology

The core approach combines transfer learning with self-learning, a form of pseudo-labeling. The process begins with a VGG16 network fine-tuned on a small initial set of labeled target data.

Overview of the self-learning pipeline incorporating transfer learning.

The model predicts labels for the unlabeled data, and the most confident predictions (e.g., the top 2-5%) are added to the labeled training pool. The model is then retrained on this merged dataset. This cycle repeats for 25 "meta-iterations." The authors evaluate this pipeline starting with both random weights and ImageNet pretrained weights across several loss functions, including standard Softmax Cross-Entropy and similarity metric learning losses (Triplet, Contrastive, and Arcface).

Experimental Setup

The approach was evaluated on three benchmarks with restricted label sets to simulate real-world scarcity:

  • SVHN (Street View House Numbers): 1,000 initially labeled images.
  • CIFAR-10: 4,000 initially labeled images.
  • Plant Village: 380 initially labeled images (just 10 per class) across 38 categories of plant leaves.

Results and Discussion

The experiments consistently showed that transfer learning acts as a universal booster for SSL. Regardless of the loss function, starting with ImageNet weights provided a significant jump in accuracy over random initialization. This pretraining provides a higher-quality starting point, mitigating the risk of confirmation bias from incorrect early pseudo-labels.

Performance comparison on CIFAR-10 across different loss functions and initialization strategies.

As shown in the CIFAR-10 results, self-learning with cross-entropy and pretrained weights achieved 83.33% accuracy, compared to 79.15% with random weights. Similar trends were observed in SVHN and the sparse Plant Village dataset. Notably, while cross-entropy generally performed best, Triplet loss proved highly competitive in extreme data scarcity scenarios, such as the Plant Village dataset, where it reached 77.80% accuracy with pretraining compared to a dismal 33.16% with random initialization.

Conclusion

The study demonstrates that standard transfer learning is a highly effective, yet often underutilized, catalyst for semi-supervised self-learning. By providing a robust feature representation from the start, transfer learning allows SSL algorithms to propagate labels more accurately. Future work could explore combinations of loss functions and modern architectures like Vision Transformers to further enhance SSL performance.

Summary updated

Abstract

Deep neural networks produce state-of-the-art results when trained on a large number of labeled examples but tend to overfit when small amounts of labeled examples are used for training. Creating a large number of labeled examples requires considerable resources, time, and effort. If labeling new data is not feasible, so-called semi-supervised learning can achieve better generalisation than purely supervised learning by employing unlabeled instances as well as labeled ones. The work presented in this paper is motivated by the observation that transfer learning provides the opportunity to potentially further improve performance by exploiting models pretrained on a similar domain. More specifically, we explore the use of transfer learning when performing semi-supervised learning using self-learning. The main contribution is an empirical evaluation of transfer learning using different combinations of similarity metric learning methods and label propagation algorithms in semi-supervised learning. We find that transfer learning always substantially improves the model's accuracy when few labeled examples are available, regardless of the type of loss used for training the neural network. This finding is obtained by performing extensive experiments on the SVHN, CIFAR10, and Plant Village image classification datasets and applying pretrained weights from Imagenet for transfer learning.
BibTeX
@inproceedings{sahito2020transfer,
  author = {Attaullah Sahito and Eibe Frank and Bernhard Pfahringer},
  title = {{Transfer of Pretrained Model Weights Substantially Improves Semi-Supervised Image Classification}},
  booktitle = {Australasian Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=SAfbqJNKHlc},
}
Figure