TL;DR
This paper tackles the challenge of overfitting in medical imaging with limited datasets by exploring novel activation functions and ensemble methods for Convolutional Neural Networks (CNNs). They found that ensembles of CNNs with diverse activation functions significantly outperform single CNNs, with custom activation functions and ensemble strategies showing particular promise.
What Problem Does This Paper Address?
Medical imaging often presents a significant challenge for deep learning models, primarily due to the "small sample size" problem. Acquiring vast datasets of medical images is frequently prohibitively expensive and labor-intensive. This scarcity of data can lead to overfitting, where models become too specialized to the training data and fail to generalize to new, unseen images. While techniques like transfer learning and data augmentation exist, building ensembles of pre-trained CNNs offers another powerful avenue to boost performance and combat overfitting. The core issue this paper addresses is how to best leverage activation functions and ensemble strategies to maximize CNN performance on these constrained medical datasets.
Methods
The researchers investigated the impact of various activation functions (AFs) on the performance of two popular CNN architectures, VGG16 and ResNet50. They tested a total of twenty AFs, including six novel ones they developed: 2D Mexican ReLU, TanELU, MeLU + GaLU, Symmetric MeLU, Symmetric GaLU, and Flexible MeLU.
Two primary ensemble methods were explored:
- Sequential Forward Floating Selection (SFFS): This method iteratively selects the best-performing classifiers to add to an ensemble, aiming for optimal performance and independence.
- Stochastic Method (Stoc): This approach involves randomly substituting activation functions within a CNN architecture from a pool of candidates, creating multiple diverse networks that are then fused.
The study also evaluated combinations of these methods and explored different configurations for the novel activation functions.
Datasets
The study utilized fifteen publicly available biomedical datasets, categorized as small to midsize. These datasets cover a range of classification tasks, including:
- Cellular imaging (e.g., Chinese hamster ovary cells, HeLa cells)
- Age classification (e.g., C. elegans muscle aging, terminal bulb aging)
- Disease detection (e.g., Lymphoma, Colorectal Cancer, Breast grading carcinoma, Laryngeal tissue)
- Tissue analysis (e.g., Liver tissue)
The paper provides a detailed table listing these datasets, their class counts, sample sizes, image types, and cross-validation protocols.
Key Insights and Ideas
- Ensembles are King: The most striking insight is the consistent superiority of ensemble methods over individual CNNs, regardless of the activation function used. This reinforces the idea that diversity within an ensemble is a powerful tool for improving generalization.
- Novel AFs Show Promise: The newly proposed activation functions, particularly MeLU variants and GaLU, demonstrated competitive or superior performance compared to established AFs like ReLU, especially when integrated into ensembles.
- Learnable AFs Shine on Small Data: The study suggests that learnable activation functions, which can adapt their parameters during training, are particularly beneficial for small to midsize datasets, as they can better capture the nuances of limited data.
- No Single "Best" AF: While some AFs performed better than others on specific architectures or datasets, there wasn't a single activation function that universally dominated across all scenarios. This highlights the importance of AF selection and ensemble strategies tailored to the specific task.
Key Results
- Ensemble Superiority: Ensemble methods consistently outperformed stand-alone CNNs across various datasets and architectures. For instance, in ResNet50, ensemble methods achieved an average accuracy of around 80-86%, while stand-alone ReLU achieved around 76%.
- Top Performers: For ResNet50, MeLU (k=8) with maxInput=255 and Splash were among the best-performing stand-alone activations, outperforming ReLU. For VGG16, MeLU (k=4) and GaLU with maxInput=255 showed strong performance.
- Stochastic Ensembles Excel: The stochastic ensemble methods (Stoc_2 and Stoc_4) generally achieved the highest accuracies, outperforming other ensemble strategies and stand-alone networks. Stoc_4, in particular, showed strong performance across multiple CNN topologies.
- 15ReLU Advantage: Even a simple ensemble of 15 ReLU networks (15ReLU) significantly outperformed the single ReLU network, demonstrating the power of ensemble averaging.
Limitations and Future Work
- Computational Cost: The study acknowledges that generating these ensembles, especially the stochastic ones, requires significant computational resources, which can be a barrier to widespread adoption.
- Generalizability to Other Modalities: While the study focused on image classification, the authors suggest future work to investigate whether the performance gains observed would transfer to other medical imaging tasks like CT/MRI analysis or tumor segmentation.
- Broader Data Exploration: The authors recommend further studies with even more diverse datasets to fully assess the performance of their proposed approach.
- Hardware Impact: The paper notes that hardware improvements can reduce inference time, but the focus of the study was on algorithmic improvements rather than hardware optimization.
Summary updated