Transferring Ensemble Representations Using Deep Convolutional Neural Networks for Small-Scale Image Classification

Shuyin Xia, Yulong Xia, Hong Yu, Qun Liu, Yueguo Luo, Guoyin Wang, Zizhong ChenIEEE Access 2019 · OpenReview

Abstract

Deep convolutional neural networks (DCNN) require a large number of training data to avoid overfitting, which makes them unsuitable for processing small-scale image datasets. Transfer learning using DCNN (TCNN) reuses pre-trained layers to generate a mid-level image representation so that the optimization of millions of CNN parameters can be avoided. In this way, the overfitting problem in small-scale data can be alleviated. However, although many public DCNNs have been trained and can be reused, existing TCNNs are formed by only a single pre-trained DCNN structure and cannot make full use of multiple structures of pre-trained DCNNs. At the same time, existing ensemble CNNs do not have sufficiently good representation ability. To address this problem, we combine the conventional ideas of ensemble CNNs and propose three ensemble TCNNs (TECNN). They are the voting method based on the combination of all TCNNs, the PickOver method by finding the optimal combination, and the weighted method by finding a weighted combination. Different from existing ensemble CNNs, the proposed methods do not need to retrain the component CNNs and generate ensemble transferring representations by transferring the pre-trained mid-level parameters. The mathematical models of those three methods are also provided. Their versions using fine-tuning are also compared in the experiments. In addition, we replace the Softmax classifier with ensemble linear classifiers in the full-connection layer. They outperform the current state-of-the-art algorithms on Caltech ImageNet and some internet image data. All this research has been released as an open-source library called Transferring Image Ensemble Representations using Deep Convolutional Neural Networks (TECNN).

Introduction

Deep Convolutional Neural Networks (DCNNs) have revolutionized computer vision, but they are notoriously data-hungry. When applied to small-scale datasets, these high-capacity models—often containing millions of parameters—tend to overfit, memorizing noise rather than learning generalizable features. This limitation makes DCNNs difficult to apply in domains where data is scarce.

Transfer Learning Fundamentals

Transfer Learning (TCNN) has become the standard solution to this problem. It involves reusing a model pre-trained on a massive dataset, such as ImageNet, and applying its learned features to a new, smaller target task. By freezing the early convolutional layers and only training the final classification layers, the number of parameters to optimize is drastically reduced.

Standard Transfer Learning (TCNN) process.

However, most researchers rely on a single architecture (e.g., just a ResNet or just a VGG). This ignores the diverse feature-extraction capabilities of different architectures. Furthermore, existing ensemble methods often require retraining the component networks, which is computationally expensive and risky when data is limited.

The TECNN Framework

The authors propose the Transferring Ensemble Convolutional Neural Network (TECNN) framework. This approach integrates multiple popular pre-trained models—specifically VGG16, VGG19, ResNet50, InceptionV3, and Xception—into a single, robust representation.

The proposed Transferring Ensemble Convolutional Neural Network (TECNN) architecture.

Instead of retraining these backbones from scratch, TECNN transfers the mid-level parameters (convolutional layers) and focuses on the "adaptation layer" (the final fully connected layers). By combining the outputs of multiple architectures, the framework creates a more comprehensive feature set than any single model could achieve.

Ensemble Strategies

The paper introduces three specific ensemble strategies to combine these transferred representations:

  1. Voting Method: A plurality-based approach where the final label is determined by the consensus of all transferred models.
  2. PickOver Method: An optimization strategy that identifies the optimal subset of models for a specific task, rather than using all of them blindly. This addresses the "more is not always better" problem, where certain architectures might introduce noise for specific datasets.
  3. Weighted Method: Assigns weights to each model based on its validation performance, allowing more reliable architectures to have a greater influence on the final classification.

Additionally, the authors experimented with replacing the standard Softmax classifier with ensemble linear classifiers (like SVMs) and tested a fine-tuning variant (TECNN+) that uses the RMSprop method to adjust only the last fully connected layer.

Experimental Results

The researchers evaluated TECNN across three distinct sources: Internet image data (scraped from Google and Baidu), Caltech256, and specific ImageNet subsets. The results consistently outperformed both standard CNNs and single-model TCNNs:

  • Internet Data: The PickOver method achieved an average accuracy improvement of 9.55% over a standard CNN and 1.65% over the best-performing single model (VGG16).
  • Caltech256: The Weighted method showed a 5.85% improvement over the most effective single TCNN (VGG19).
  • Fine-tuning: The fine-tuned versions (TECNN+) generally achieved higher generalizability, though the authors noted that fine-tuning too many layers on very small data could still trigger overfitting.

Conclusion

TECNN provides a powerful framework for small-scale image classification by leveraging the diversity of multiple DCNN architectures without the need for expensive retraining. By freezing convolutional layers and optimizing ensemble strategies, it effectively mitigates overfitting while reaching state-of-the-art accuracy. The authors have released their work as an open-source library to facilitate further research in small-data deep learning.

Summary updated

Abstract

The deep convolutional neural networks (DCNN) require large number of training data to avoid overfitting, which makes it unsuitable for processing small-scale image datasets. The transfer learning using DCNN (TCNN) reuses pre-trained layers to generate a mid-level image representation so that the optimization of more than millions CNN parameters can be avoided. By this way, overfitting problem in small-scale data can be alleviated. However, although now many public DCNNs have been trained and can be reused, the existing TCNNs are formed by only a single pre-trained DCNN structure and cannot make full use of multiple structures of pre-trained DCNNs. At the same time, the existing ensemble CNNs have not enough good representation ability. To address this problem, we combine the conventional ideas of ensemble CNNs and propose three ensemble TCNNs (TECNN). They are the voting method based on the combination of all TCNNs, the PickOver method by finding the optimal combination, and weighted method by finding weighted combination. Different from the existing ensemble CNNs, the proposed methods do not need to retrain the component CNNs and generate ensemble transferring representations by transferring the pre-trained mid-level parameters. The mathematical models of those three methods are also provided. Their versions of using fine-tuning are also compared in the experiments. In addition, we replace the Softmax classifier with ensemble linear classifiers in the full-connection layer. They outperform the current state of the art algorithms on Caltech ImageNet and some internet image data. All this research has released as an open source library called Transferring Image Ensemble Representations using Deep Convolutional Neural Networks (TECNN). The source codes and relevant datasets in different versions are available from: http://www.cquptshuyinxia.com/TECNN.html.
BibTeX
@inproceedings{xia2019transferring,
  author = {Shuyin Xia and Yulong Xia and Hong Yu and Qun Liu and Yueguo Luo and Guoyin Wang and Zizhong Chen},
  title = {{Transferring Ensemble Representations Using Deep Convolutional Neural Networks for Small-Scale Image Classification}},
  booktitle = {IEEE Access},
  year = {2019},
  url = {https://openreview.net/forum?id=UJAf5YRHS0},
}
Figure