Ensemble of Convolutional Neural Networks Trained with Different\n Activation Functions

Gianluca Maguolo, Loris Nanni, Stefano Ghidoni

This paper explores a simple yet powerful idea: if different activation functions have different strengths, why not use an ensemble of them? Researchers from the University of Padua demonstrate that combining Convolutional Neural Networks (CNNs) trained with a variety of activation functions significantly boosts performance, particularly on the small-to-medium-sized datasets common in biomedical imaging.

TL;DR

The authors propose an ensemble approach where multiple CNNs are trained using different activation functions, outperforming any single-function model. They also introduce Mexican ReLU (MeLU), a novel learnable activation function that uses "Mexican hat" basis functions to approximate complex non-linearities with high stability.

What Problem Does This Paper Address?

Activation functions (AFs) like ReLU, ELU, and Swish are the "gatekeepers" of neural networks, determining how information flows and how gradients are backpropagated. While ReLU is the industry standard due to its simplicity and speed, it isn't always the best choice for every dataset—especially smaller biomedical datasets where the risk of vanishing gradients or overfitting is high.

Existing "learnable" activation functions (which change shape during training) often struggle with optimization stability. For instance, modifying a parameter in some functions might change the activation's behavior across the entire input range, potentially disrupting previously learned features. The authors sought a way to gain the representational power of learnable activations without the training instability.

Methods

The paper introduces two main contributions:

  1. The Ensemble Strategy: Instead of picking one "best" activation function, the authors train multiple instances of standard architectures (VGG16 and ResNet50). Each instance uses a different AF (ReLU, Leaky ReLU, ELU, SELU, PReLU, SReLU, or APLU). They then combine these models using a "sum rule" ensemble, which averages the output scores.
  2. Mexican ReLU (MeLU): This is a new piecewise linear activation function. It is defined as the sum of a Parametric ReLU (PReLU) and several "Mexican hat" functions.
  3. Local Support: Unlike other learnable functions, each "Mexican hat" component only affects a small, specific interval of the input. This means the optimization process can "fine-tune" the activation's shape in one area without messing up its behavior elsewhere.
  4. ReLU Initialization: MeLU can be initialized so that it starts exactly like a standard ReLU, making it highly compatible with transfer learning from pre-trained models.

Datasets

The authors evaluated their approach on 13 diverse biomedical datasets, including:

  • CH & HE: Subcellular structures in Hamster Ovary and HeLa cells.
  • RN: Fly cell images.
  • CO: Histological images of human colorectal cancer.
  • BGR & LAR: Breast grading and laryngeal cancer datasets.
  • TB, LY, MA, LG, LA: Various aging and tissue-specific datasets (e.g., muscle aging, liver gender).

These datasets are challenging because they often have a limited number of samples (ranging from 200 to 5,000), making model robustness critical.

Key Insights and Ideas

  • Diversity is Key: The core "aha moment" is that the errors made by a ReLU-based network are often different from those made by an ELU-based network. By ensembling them, these idiosyncratic errors cancel out, leading to a more robust classifier.
  • Optimization via Localization: The MeLU function’s use of basis functions with limited support is a clever trick. In optimization terms, it makes the "loss landscape" easier to navigate because parameters are less entangled.
  • Superiority over ReLU Ensembles: Crucially, the authors show that an ensemble of networks with different activation functions (ACT ENS) performs better than an ensemble of the same size using only ReLU networks (RELU ENS). This proves that the benefit comes from the diversity of the activations, not just the increased number of parameters.

Key Results

  • Statistical Significance: The proposed ensemble (eENS) outperformed every single standalone activation function with a p-value of less than 0.01.
  • MeLU Performance: As a standalone function, MeLU (specifically with $k=4$ parameters) was frequently the top performer among all tested activations, showing its strength as a general-purpose AF.
  • Accuracy Boosts: On the ResNet50 architecture, the ensemble achieved an average accuracy of 87.06% across all datasets, compared to 84.54% for a standard ReLU-based ResNet50.
  • VGG16 Success: Similar gains were seen with VGG16, where the ensemble reached 89.68% average accuracy versus 87.33% for the ReLU baseline.

Limitations and Future Work

  • Computational Overhead: The primary drawback is cost. Training and running an ensemble of 24 networks is significantly more memory-intensive and slower than a single model.
  • Implementation Speed: While MeLU is mathematically sound, standard ReLU is still much faster in practice because it is natively optimized in deep learning frameworks like TensorFlow and PyTorch. The authors' MATLAB implementation of MeLU is currently slower.
  • Future Directions: The authors suggest exploring "distillation" (training a single small network to mimic the ensemble) to get the accuracy benefits without the computational cost. They also plan to test if these ensembles allow very small CNNs to compete with much larger, single-model architectures.

Summary updated

Abstract

Activation functions play a vital role in the training of Convolutional\nNeural Networks. For this reason, to develop efficient and performing functions\nis a crucial problem in the deep learning community. Key to these approaches is\nto permit a reliable parameter learning, avoiding vanishing gradient problems.\nThe goal of this work is to propose an ensemble of Convolutional Neural\nNetworks trained using several different activation functions. Moreover, a\nnovel activation function is here proposed for the first time. Our aim is to\nimprove the performance of Convolutional Neural Networks in small/medium size\nbiomedical datasets. Our results clearly show that the proposed ensemble\noutperforms Convolutional Neural Networks trained with standard ReLU as\nactivation function. The proposed ensemble outperforms with a p-value of 0.01\neach tested stand-alone activation function; for reliable performance\ncomparison we have tested our approach in more than 10 datasets, using two\nwell-known Convolutional Neural Network: Vgg16 and ResNet50. MATLAB code used\nhere will be available at https://github.com/LorisNanni.\n
BibTeX
@inproceedings{maguolo2020ensemble,
  author = {Gianluca Maguolo and Loris Nanni and Stefano Ghidoni},
  title = {{Ensemble of Convolutional Neural Networks Trained with Different\textbackslash\{\}n Activation Functions}},
  booktitle = {Expert},
  year = {2020},
  doi = {10.1016/j.eswa.2020.114048},
}
Figure