Effectiveness of Self Normalizing Neural Networks for Text Classification

Avinash Madasu, Vijjini Anvesh Rao

TL;DR

This paper introduces Self-Normalizing Convolutional Neural Networks (SCNN) for text classification, adapting the principles of Self-Normalizing Neural Networks (SNNs) to the unique constraints of Natural Language Processing. By swapping the standard SELU activation for ELU to accommodate non-normalized word embeddings, the authors created a model that achieves state-of-the-art performance with significantly fewer parameters than standard CNN architectures.

What Problem Does This Paper Address?

Deep neural networks often struggle with stability during training. Standard architectures like Convolutional Neural Networks (CNNs) typically rely on Batch Normalization to ensure that neuron activations don't drift toward extreme values (vanishing or exploding gradients).

While Self-Normalizing Neural Networks (SNNs) were previously proposed to solve this by using the SELU activation function to automatically converge activations to a fixed mean and variance, they were primarily designed for computer vision. In NLP, we use word embeddings, which represent semantic relationships through their spatial distribution. Normalizing these embeddings (a requirement for standard SNNs) destroys these semantic relationships. The authors sought to bring the benefits of self-normalization to text classification without breaking the underlying logic of word vectors.

Methods

The authors propose the SCNN architecture, which incorporates several key components of the SNN framework but with a critical modification for text:

  1. ELU instead of SELU: The standard SNN uses SELU, which includes a scaling parameter ($\lambda > 1$). If inputs aren't perfectly normalized (like word embeddings), this $\lambda$ causes the mean and variance to shift and magnify across layers. The authors use Exponential Linear Units (ELU) instead, which lack the $\lambda$ parameter and push activations toward a unit mean even when inputs aren't normalized.
  2. LeCun Normal Initialization: Weights are initialized from a distribution centered at 0 with a specific standard deviation ($\sqrt{1/in}$), which is essential for maintaining the self-normalizing property.
  3. Alpha Dropout: Unlike standard dropout, which sets weights to zero and shifts the mean, Alpha Dropout randomly sets activations to negative saturation values. This preserves the mean and variance of the network during training.
  4. Architecture: A standard 1D-CNN structure (using kernels of sizes 3, 4, and 5) but replacing ReLU/Batch-Norm with the SNN components mentioned above.

Datasets

The model was evaluated on six diverse text classification benchmarks:

  • MR: Movie Reviews (Sentiment analysis).
  • SO: Subjectivity/Objectivity dataset.
  • IMDB: Large-scale movie review dataset.
  • TREC: Question classification (6 categories).
  • CR: Customer Reviews.
  • MPQA: Opinion/Sentiment dataset.

Key Insights and Ideas

  • The Embedding Constraint: You cannot treat text like pixels. In Computer Vision, normalizing pixel values is standard. In NLP, normalizing a word vector changes its "meaning" relative to other words. This is why a "pure" SNN (using SELU) performs worse on text than the proposed SCNN (using ELU).
  • Solving the "Dying ReLU" Problem: Standard CNNs use ReLU, which zeroes out all negative values. This causes the network to "lose" information contained in the negative values of pre-trained word vectors (like Word2Vec). ELU and SELU allow negative values to pass through, preserving more semantic information.
  • Efficiency via Architecture: By making the network "self-normalizing" through its activation and initialization, you eliminate the need for Batch Normalization layers, reducing the total parameter count and computational overhead.

Key Results

The SCNN was compared against a standard "Static CNN" (a popular baseline by Yoon Kim) and a "Short CNN" (a standard CNN with the same number of parameters as the SCNN).

  • Parameter Efficiency: The SCNN used roughly 30% fewer parameters than the standard Static CNN (e.g., ~254k vs ~362k parameters on several datasets).
  • Superior Performance: Despite having fewer parameters, SCNN outperformed the standard Static CNN on the IMDB, CR, and MPQA datasets.
  • Head-to-Head: When compared to a standard CNN with the exact same number of parameters (Short CNN), the SCNN won across every single dataset, often by a margin of 2-4% in accuracy or F1-score.
  • ELU vs. SELU: The results confirmed that for text, ELU is a better choice than SELU, consistently yielding higher accuracy because it handles non-normalized embeddings more gracefully.

Limitations and Future Work

  • Architectural Simplicity: The paper focuses on relatively shallow CNN architectures. The authors note that future work should explore how SCNN principles scale to much deeper architectures.
  • Scope: The study is limited to 1D-CNNs for classification. The authors suggest that applying these self-normalizing principles to Recurrent Neural Networks (RNNs) is a promising next step.
  • Embedding Types: The study primarily uses pre-trained Word2Vec embeddings; it remains to be seen how SCNN interacts with dynamic, contextual embeddings like BERT or RoBERTa.

Summary updated

Abstract

Self Normalizing Neural Networks(SNN) proposed on Feed Forward Neural Networks(FNN) outperform regular FNN architectures in various machine learning tasks. Particularly in the domain of Computer Vision, the activation function Scaled Exponential Linear Units (SELU) proposed for SNNs, perform better than other non linear activations such as ReLU. The goal of SNN is to produce a normalized output for a normalized input. Established neural network architectures like feed forward networks and Convolutional Neural Networks(CNN) lack the intrinsic nature of normalizing outputs. Hence, requiring additional layers such as Batch Normalization. Despite the success of SNNs, their characteristic features on other network architectures like CNN haven't been explored, especially in the domain of Natural Language Processing. In this paper we aim to show the effectiveness of proposed, Self Normalizing Convolutional Neural Networks(SCNN) on text classification. We analyze their performance with the standard CNN architecture used on several text classification datasets. Our experiments demonstrate that SCNN achieves comparable results to standard CNN model with significantly fewer parameters. Furthermore it also outperforms CNN with equal number of parameters.
BibTeX
@inproceedings{madasu2019effectiveness,
  author = {Avinash Madasu and Vijjini Anvesh Rao},
  title = {{Effectiveness of Self Normalizing Neural Networks for Text Classification}},
  booktitle = {Cicling},
  year = {2019},
  doi = {10.1007/978-3-031-24340-0\_31},
}
Figure