Comparison of Different Convolutional Neural Network Activation Functions and Methods for Building Ensembles for Small to Midsize Medical Data Sets

Loris Nanni, Sheryl Brahnam, Michelangelo Paci, Stefano GhidoniSensors 2022 · DOI

TL;DR

This paper tackles the challenge of overfitting in medical imaging with limited datasets by exploring novel activation functions and ensemble methods for Convolutional Neural Networks (CNNs). They found that ensembles of CNNs with diverse activation functions significantly outperform single CNNs, with custom activation functions and ensemble strategies showing particular promise.

What Problem Does This Paper Address?

Medical imaging often presents a significant challenge for deep learning models, primarily due to the "small sample size" problem. Acquiring vast datasets of medical images is frequently prohibitively expensive and labor-intensive. This scarcity of data can lead to overfitting, where models become too specialized to the training data and fail to generalize to new, unseen images. While techniques like transfer learning and data augmentation exist, building ensembles of pre-trained CNNs offers another powerful avenue to boost performance and combat overfitting. The core issue this paper addresses is how to best leverage activation functions and ensemble strategies to maximize CNN performance on these constrained medical datasets.

Methods

The researchers investigated the impact of various activation functions (AFs) on the performance of two popular CNN architectures, VGG16 and ResNet50. They tested a total of twenty AFs, including six novel ones they developed: 2D Mexican ReLU, TanELU, MeLU + GaLU, Symmetric MeLU, Symmetric GaLU, and Flexible MeLU.

Two primary ensemble methods were explored:

  1. Sequential Forward Floating Selection (SFFS): This method iteratively selects the best-performing classifiers to add to an ensemble, aiming for optimal performance and independence.
  2. Stochastic Method (Stoc): This approach involves randomly substituting activation functions within a CNN architecture from a pool of candidates, creating multiple diverse networks that are then fused.

The study also evaluated combinations of these methods and explored different configurations for the novel activation functions.

Datasets

The study utilized fifteen publicly available biomedical datasets, categorized as small to midsize. These datasets cover a range of classification tasks, including:

  • Cellular imaging (e.g., Chinese hamster ovary cells, HeLa cells)
  • Age classification (e.g., C. elegans muscle aging, terminal bulb aging)
  • Disease detection (e.g., Lymphoma, Colorectal Cancer, Breast grading carcinoma, Laryngeal tissue)
  • Tissue analysis (e.g., Liver tissue)

The paper provides a detailed table listing these datasets, their class counts, sample sizes, image types, and cross-validation protocols.

Key Insights and Ideas

  • Ensembles are King: The most striking insight is the consistent superiority of ensemble methods over individual CNNs, regardless of the activation function used. This reinforces the idea that diversity within an ensemble is a powerful tool for improving generalization.
  • Novel AFs Show Promise: The newly proposed activation functions, particularly MeLU variants and GaLU, demonstrated competitive or superior performance compared to established AFs like ReLU, especially when integrated into ensembles.
  • Learnable AFs Shine on Small Data: The study suggests that learnable activation functions, which can adapt their parameters during training, are particularly beneficial for small to midsize datasets, as they can better capture the nuances of limited data.
  • No Single "Best" AF: While some AFs performed better than others on specific architectures or datasets, there wasn't a single activation function that universally dominated across all scenarios. This highlights the importance of AF selection and ensemble strategies tailored to the specific task.

Key Results

  • Ensemble Superiority: Ensemble methods consistently outperformed stand-alone CNNs across various datasets and architectures. For instance, in ResNet50, ensemble methods achieved an average accuracy of around 80-86%, while stand-alone ReLU achieved around 76%.
  • Top Performers: For ResNet50, MeLU (k=8) with maxInput=255 and Splash were among the best-performing stand-alone activations, outperforming ReLU. For VGG16, MeLU (k=4) and GaLU with maxInput=255 showed strong performance.
  • Stochastic Ensembles Excel: The stochastic ensemble methods (Stoc_2 and Stoc_4) generally achieved the highest accuracies, outperforming other ensemble strategies and stand-alone networks. Stoc_4, in particular, showed strong performance across multiple CNN topologies.
  • 15ReLU Advantage: Even a simple ensemble of 15 ReLU networks (15ReLU) significantly outperformed the single ReLU network, demonstrating the power of ensemble averaging.

Limitations and Future Work

  • Computational Cost: The study acknowledges that generating these ensembles, especially the stochastic ones, requires significant computational resources, which can be a barrier to widespread adoption.
  • Generalizability to Other Modalities: While the study focused on image classification, the authors suggest future work to investigate whether the performance gains observed would transfer to other medical imaging tasks like CT/MRI analysis or tumor segmentation.
  • Broader Data Exploration: The authors recommend further studies with even more diverse datasets to fully assess the performance of their proposed approach.
  • Hardware Impact: The paper notes that hardware improvements can reduce inference time, but the focus of the study was on algorithmic improvements rather than hardware optimization.

Summary updated

Abstract

CNNs and other deep learners are now state-of-the-art in medical imaging research. However, the small sample size of many medical data sets dampens performance and results in overfitting. In some medical areas, it is simply too labor-intensive and expensive to amass images numbering in the hundreds of thousands. Building Deep CNN ensembles of pre-trained CNNs is one powerful method for overcoming this problem. Ensembles combine the outputs of multiple classifiers to improve performance. This method relies on the introduction of diversity, which can be introduced on many levels in the classification workflow. A recent ensembling method that has shown promise is to vary the activation functions in a set of CNNs or within different layers of a single CNN. This study aims to examine the performance of both methods using a large set of twenty activations functions, six of which are presented here for the first time: 2D Mexican ReLU, TanELU, MeLU + GaLU, Symmetric MeLU, Symmetric GaLU, and Flexible MeLU. The proposed method was tested on fifteen medical data sets representing various classification tasks. The best performing ensemble combined two well-known CNNs (VGG16 and ResNet50) whose standard ReLU activation layers were randomly replaced with another. Results demonstrate the superiority in performance of this approach.
BibTeX
@inproceedings{nanni2022comparison,
  author = {Loris Nanni and Sheryl Brahnam and Michelangelo Paci and Stefano Ghidoni},
  title = {{Comparison of Different Convolutional Neural Network Activation Functions and Methods for Building Ensembles for Small to Midsize Medical Data Sets}},
  booktitle = {Sensors},
  year = {2022},
  doi = {10.3390/s22166129},
}
Figure