TL;DR
This paper introduces a novel method for designing Convolutional Neural Networks (CNNs) by stochastically replacing activation layers with a mix of "static" and "dynamic" functions. This approach leads to improved performance, especially when these diverse models are combined into ensembles, outperforming standard ReLU-based networks on various image classification and segmentation tasks.
What Problem Does This Paper Address?
Deep learning models, particularly CNNs, have achieved remarkable success in tasks like image classification and segmentation. A critical component of these networks is the activation function, which introduces non-linearity. While various "static" (fixed) and "dynamic" (learnable parameters) activation functions exist, choosing the optimal one for a specific task remains challenging. Dynamic functions often offer better performance but come with increased computational cost and a higher risk of overfitting. This paper aims to bridge this gap by proposing a method that leverages the strengths of both static and dynamic functions, creating diverse network architectures that can be effectively combined into ensembles to boost overall performance.
Methods
The core of this work lies in a novel model design strategy that replaces each activation layer in a pre-trained CNN with a randomly selected activation function from a predefined set. This set includes popular static functions like ReLU, Leaky ReLU, ELU, and dynamic functions like PReLU, SReLU, APLU, MeLU, and GaLU. The selection process is stochastic, meaning that for each layer, a different activation function can be chosen, leading to a unique network architecture.
The paper then utilizes this "stochastic replacement" to generate multiple network variants. These variants are subsequently fine-tuned on specific datasets. To further enhance performance, these diverse models are combined into ensembles using a sum rule, where the outputs of the softmax layers are summed, and the final decision is made via an argmax function. This ensemble approach leverages the inherent instability of neural network training to create partially independent classifiers whose combined predictions lead to improved accuracy and robustness.
Datasets
The proposed method was evaluated on a comprehensive set of datasets:
- Image Classification: 13 medical image classification datasets were used, including datasets for Chinese hamster ovary cells, HeLa cells, and human colorectal cancer, among others.
- Image Segmentation: 11 skin segmentation datasets were utilized, such as Compaq, UChile DB-skin, and ECU Face and Skin Detection.
For image classification, models were fine-tuned on the training set of each problem. For skin segmentation, a single training set of 2000 images from the ECU dataset was used, with other datasets serving for testing.
Key Insights and Ideas
- Diversity through Stochasticity: Randomly replacing activation layers is a powerful way to introduce diversity into CNN architectures, even when starting from the same base model. This diversity is key to the success of ensemble methods.
- Ensembles Trump Stand-alone: Across both image classification and segmentation tasks, ensemble models consistently outperformed individual stand-alone models, highlighting the benefit of combining diverse network architectures.
- Dynamic Functions Offer Potential, but Ensembles are Key: While some dynamic activation functions (like wMeLU and MeLU) showed strong performance as stand-alone models, outperforming the baseline ReLU, their true power was unleashed when integrated into ensembles.
- MaxInput Matters: The
maxInputparameter, particularly for dynamic activation functions, showed a significant impact. For skin segmentation, settingmaxInputto 1 generally yielded better results than 255. - Stochastic Ensembles Excel: Ensembles created using stochastic layer replacement (e.g., FusRan20) achieved top performance, demonstrating the effectiveness of this approach in creating robust and accurate models.
Key Results
- Image Classification: Ensembles consistently outperformed stand-alone models. The
FusRan20ensemble, a fusion of 20 stochastic models, achieved the highest rank. Several stand-alone methods, particularlywMeLU(255)andMeLU(255), significantly outperformed the baseline ReLU. - Image Segmentation: Ensembles also dominated stand-alone models.
FusAR20was the best-ranked method, with lighter ensembles likeFusAct3andFusAct10also showing strong performance. The proposed methods outperformed the standard DeepLabv3+ with ReLU. - Statistical Significance: Wilcoxon signed rank tests confirmed the statistical significance of the performance gains achieved by the proposed ensemble methods over baselines and other approaches. For instance, stochastic ensembles like
FusRan10outperformed their non-stochastic counterparts (FusAct10) with notable p-values. - Efficiency: Using a Titan Xp, an ensemble of 20 CNNs could classify more than two images per second, indicating a reasonable trade-off between performance and inference speed.
Limitations and Future Work
- Computational Cost: The paper acknowledges that studying ensembles of CNNs requires significant computational resources, which can be a limitation for extensive experimentation.
- Model Scope: The current study primarily focused on ResNet50 and DeepLabv3+ architectures. The authors suggest as future work to extend this method to a broader range of models, including lighter architectures suitable for mobile devices.
- Parameter Tuning: While the paper used parameter settings suggested by original authors for activation functions, further exploration of hyperparameter tuning for these functions within the stochastic framework could yield even better results.
Summary updated