Abstract
Deep neural networks often face stability challenges during training, typically addressed by Batch Normalization. While Self-Normalizing Neural Networks (SNNs) using SELU activations provide an alternative, they are optimized for normalized inputs, making them ill-suited for Natural Language Processing (NLP) where word embeddings carry semantic meaning in their spatial distribution. This paper introduces the Self-Normalizing Convolutional Neural Networks (SCNN), which adapts SNN principles for text by utilizing Exponential Linear Units (ELU). Our results demonstrate that SCNN achieves state-of-the-art performance with significantly fewer parameters than standard architectures.
Introduction
In text classification, standard Convolutional Neural Networks (CNNs) rely on Batch Normalization to prevent vanishing or exploding gradients. However, the requirement for normalized inputs in traditional SNNs poses a problem for NLP: normalizing word embeddings destroys the semantic relationships they represent. The authors propose SCNN to bring the stability of self-normalization to text classification without compromising the integrity of word vectors.
The SCNN Architecture
The SCNN architecture modifies the standard SNN framework to accommodate the unique constraints of text data. Key components include:
- ELU Activation: Unlike SELU, which includes a scaling parameter ($\lambda > 1$) that can magnify shifts in non-normalized inputs, ELU pushes activations toward a unit mean more gracefully, preserving the semantic information in word embeddings.
- LeCun Normal Initialization: Weights are initialized from a distribution centered at 0 with a standard deviation of $\sqrt{1/in}$, a prerequisite for maintaining self-normalizing properties.
- Alpha Dropout: To preserve the mean and variance of the network during training, Alpha Dropout is used instead of standard dropout, setting activations to negative saturation values rather than zero.
- 1D-CNN Structure: The model employs a standard 1D-CNN structure with kernel sizes of 3, 4, and 5, replacing ReLU and Batch Normalization with the aforementioned SNN components.
Experimental Results
The SCNN was evaluated against six benchmarks: MR, SO, IMDB, TREC, CR, and MPQA. It was compared to a "Static CNN" (Yoon Kim baseline) and a "Short CNN" (a standard CNN matched for parameter count).
Performance and Efficiency
SCNN demonstrated remarkable parameter efficiency, utilizing approximately 30% fewer parameters than the standard Static CNN (e.g., ~254k vs ~362k). Despite this reduction, SCNN outperformed the Static CNN on the IMDB, CR, and MPQA datasets. When compared to the parameter-matched Short CNN, SCNN was superior across every dataset, often by a margin of 2-4% in accuracy or F1-score.
The ELU Advantage
The study confirmed that ELU is a superior choice for text classification over SELU. Because ELU handles non-normalized embeddings more effectively, it consistently yielded higher accuracy. Furthermore, by allowing negative values to pass through (unlike ReLU), ELU preserves more semantic information from pre-trained word vectors like Word2Vec.
Conclusion
SCNN provides a robust, efficient framework for text classification by successfully adapting self-normalization principles for non-normalized word embeddings. By eliminating the need for Batch Normalization, SCNN reduces computational overhead while improving performance. Future research will explore the scalability of these principles in deeper architectures and their application to Recurrent Neural Networks (RNNs).
Summary updated