Introduction: The Robustness Challenge in CNNs
Modern Convolutional Neural Networks (CNNs) predominantly utilize element-level activation (ElementAct) functions, such as ReLU, SiLU, or GELU. These functions operate independently on each pixel or feature. While effective for learning complex patterns, their ability to handle noise—robustness—is often inconsistent. Robustness in these functions typically relies on "saturation," where small input perturbations do not affect the output. However, if an input image lacks sufficient features in a saturated state, the network becomes highly sensitive to noise.
While Layer Normalization (LN) can stabilize these states and enhance robustness, it often degrades performance on clean datasets for CNNs. This degradation stems from the "homogeneity" problem, where LN makes internal representations of different images too similar, reducing the network's discriminative power.
Methodology: The LayerAct Mechanism
To address these limitations, the authors propose LayerAct, a novel activation mechanism that decouples the "gating" of an activation from its "magnitude."
In a standard ElementAct setup, the activation $a_i$ is calculated as: $$a_i = y_i \cdot s(y_i)$$ where the input $y_i$ determines both the value and the scale function $s$.
In contrast, LayerAct calculates the activation as: $$a_i = y_i \cdot s(n_i)$$ where $n_i$ is the layer-normalized version of the input.
By using the normalized input ($n_i$) to control the gate, the activation becomes significantly more stable and robust to input shifts. Crucially, because the gate is multiplied by the original input ($y_i$), the unique statistical signature of the image is preserved, preventing the homogeneity issues associated with standard LN. The authors introduced two primary variants: LA-SiLU (using Sigmoid) and LA-HardSiLU (using a computationally efficient Hard Sigmoid).
Key Insights: Decoupling and Stability
The core innovation of LayerAct is the realization that stability can be derived from normalized statistics without sacrificing the discriminative detail of raw data. This approach effectively breaks the traditional trade-off between robustness and accuracy. Furthermore, LayerAct achieves "one-sided saturation" relative to layer statistics rather than a fixed zero point, which naturally promotes training stability. Unlike standard LN which often replaces Batch Normalization (BN), LayerAct is designed to complement BN, using BN for training stability and LayerAct for robust activation.
Key Results and Performance
LayerAct was evaluated across several benchmarks, including CIFAR-10/100, ImageNet, and their corrupted counterparts (CIFAR-10-C, ImageNet-C), as well as medical imaging tasks.
- Robustness: On the CIFAR-10-C benchmark using ResNet20, LA-SiLU achieved a mean accuracy of 70.4%, outperforming ReLU (68.8%) and SiLU (69.0%).
- Large-Scale Performance: On ImageNet-C with ResNet50, LA-SiLU reached 45.29% accuracy, compared to 43.75% for ReLU.
- Clean Accuracy: Notably, LayerAct does not sacrifice clean performance. LA-SiLU improved clean ImageNet accuracy for ResNet50 from 77.71% to 78.62%.
- Noise Stability: Empirical analysis shows that LayerAct significantly reduces the mean of fluctuation under noise compared to standard activations.
In medical imaging (nuclei segmentation), LayerAct improved the Intersection over Union (IoU) in both U-Net and UNet++ architectures, demonstrating its versatility beyond classification.
Limitations and Future Outlook
While LayerAct shows significant promise, there are areas for further study. The calculation of mean and variance for every activation layer introduces a minor computational overhead, though this is generally negligible in large-scale CNNs. Future research could explore the application of LayerAct in Vision Transformers (ViTs) and conduct a more granular sensitivity analysis of the stability constant ($\alpha$) across diverse architectures.
Summary updated