Stochastic Selection of Activation Layers for Convolutional Neural Networks

Loris Nanni, Alessandra Lumini, Stefano Ghidoni, Gianluca MaguoloSensors 2020 · DOI

TL;DR

This paper introduces a novel method for designing Convolutional Neural Networks (CNNs) by stochastically replacing activation layers with a mix of "static" and "dynamic" functions. This approach leads to improved performance, especially when these diverse models are combined into ensembles, outperforming standard ReLU-based networks on various image classification and segmentation tasks.

What Problem Does This Paper Address?

Deep learning models, particularly CNNs, have achieved remarkable success in tasks like image classification and segmentation. A critical component of these networks is the activation function, which introduces non-linearity. While various "static" (fixed) and "dynamic" (learnable parameters) activation functions exist, choosing the optimal one for a specific task remains challenging. Dynamic functions often offer better performance but come with increased computational cost and a higher risk of overfitting. This paper aims to bridge this gap by proposing a method that leverages the strengths of both static and dynamic functions, creating diverse network architectures that can be effectively combined into ensembles to boost overall performance.

Methods

The core of this work lies in a novel model design strategy that replaces each activation layer in a pre-trained CNN with a randomly selected activation function from a predefined set. This set includes popular static functions like ReLU, Leaky ReLU, ELU, and dynamic functions like PReLU, SReLU, APLU, MeLU, and GaLU. The selection process is stochastic, meaning that for each layer, a different activation function can be chosen, leading to a unique network architecture.

The paper then utilizes this "stochastic replacement" to generate multiple network variants. These variants are subsequently fine-tuned on specific datasets. To further enhance performance, these diverse models are combined into ensembles using a sum rule, where the outputs of the softmax layers are summed, and the final decision is made via an argmax function. This ensemble approach leverages the inherent instability of neural network training to create partially independent classifiers whose combined predictions lead to improved accuracy and robustness.

Datasets

The proposed method was evaluated on a comprehensive set of datasets:

  • Image Classification: 13 medical image classification datasets were used, including datasets for Chinese hamster ovary cells, HeLa cells, and human colorectal cancer, among others.
  • Image Segmentation: 11 skin segmentation datasets were utilized, such as Compaq, UChile DB-skin, and ECU Face and Skin Detection.

For image classification, models were fine-tuned on the training set of each problem. For skin segmentation, a single training set of 2000 images from the ECU dataset was used, with other datasets serving for testing.

Key Insights and Ideas

  • Diversity through Stochasticity: Randomly replacing activation layers is a powerful way to introduce diversity into CNN architectures, even when starting from the same base model. This diversity is key to the success of ensemble methods.
  • Ensembles Trump Stand-alone: Across both image classification and segmentation tasks, ensemble models consistently outperformed individual stand-alone models, highlighting the benefit of combining diverse network architectures.
  • Dynamic Functions Offer Potential, but Ensembles are Key: While some dynamic activation functions (like wMeLU and MeLU) showed strong performance as stand-alone models, outperforming the baseline ReLU, their true power was unleashed when integrated into ensembles.
  • MaxInput Matters: The maxInput parameter, particularly for dynamic activation functions, showed a significant impact. For skin segmentation, setting maxInput to 1 generally yielded better results than 255.
  • Stochastic Ensembles Excel: Ensembles created using stochastic layer replacement (e.g., FusRan20) achieved top performance, demonstrating the effectiveness of this approach in creating robust and accurate models.

Key Results

  • Image Classification: Ensembles consistently outperformed stand-alone models. The FusRan20 ensemble, a fusion of 20 stochastic models, achieved the highest rank. Several stand-alone methods, particularly wMeLU(255) and MeLU(255), significantly outperformed the baseline ReLU.
  • Image Segmentation: Ensembles also dominated stand-alone models. FusAR20 was the best-ranked method, with lighter ensembles like FusAct3 and FusAct10 also showing strong performance. The proposed methods outperformed the standard DeepLabv3+ with ReLU.
  • Statistical Significance: Wilcoxon signed rank tests confirmed the statistical significance of the performance gains achieved by the proposed ensemble methods over baselines and other approaches. For instance, stochastic ensembles like FusRan10 outperformed their non-stochastic counterparts (FusAct10) with notable p-values.
  • Efficiency: Using a Titan Xp, an ensemble of 20 CNNs could classify more than two images per second, indicating a reasonable trade-off between performance and inference speed.

Limitations and Future Work

  • Computational Cost: The paper acknowledges that studying ensembles of CNNs requires significant computational resources, which can be a limitation for extensive experimentation.
  • Model Scope: The current study primarily focused on ResNet50 and DeepLabv3+ architectures. The authors suggest as future work to extend this method to a broader range of models, including lighter architectures suitable for mobile devices.
  • Parameter Tuning: While the paper used parameter settings suggested by original authors for activation functions, further exploration of hyperparameter tuning for these functions within the stochastic framework could yield even better results.

Summary updated

Abstract

In recent years, the field of deep learning has achieved considerable success in pattern recognition, image segmentation, and many other classification fields. There are many studies and practical applications of deep learning on images, video, or text classification. Activation functions play a crucial role in discriminative capabilities of the deep neural networks and the design of new "static" or "dynamic" activation functions is an active area of research. The main difference between "static" and "dynamic" functions is that the first class of activations considers all the neurons and layers as identical, while the second class learns parameters of the activation function independently for each layer or even each neuron. Although the "dynamic" activation functions perform better in some applications, the increased number of trainable parameters requires more computational time and can lead to overfitting. In this work, we propose a mixture of "static" and "dynamic" activation functions, which are stochastically selected at each layer. Our idea for model design is based on a method for changing some layers along the lines of different functional blocks of the best performing CNN models, with the aim of designing new models to be used as stand-alone networks or as a component of an ensemble. We propose to replace each activation layer of a CNN (usually a ReLU layer) by a different activation function stochastically drawn from a set of activation functions: in this way, the resulting CNN has a different set of activation function layers.
BibTeX
@inproceedings{nanni2020stochastic,
  author = {Loris Nanni and Alessandra Lumini and Stefano Ghidoni and Gianluca Maguolo},
  title = {{Stochastic Selection of Activation Layers for Convolutional Neural Networks}},
  booktitle = {Sensors},
  year = {2020},
  doi = {10.3390/s20061626},
}
Figure