Introduction
Training deep neural networks, particularly Recurrent Neural Networks (RNNs), is notoriously difficult due to vanishing and exploding gradients. A key factor in stable training is ensuring that the inputs to each layer are "zero-centered" (having a mean of zero). When activations have a non-zero mean—as is the case with the popular Rectified Linear Unit (ReLU), which is always non-negative—it introduces a bias in weight updates that slows down convergence.
While techniques like Batch Normalization (BN) and Layer Normalization solve this by explicitly re-centering activations, they add computational overhead and can be complex to implement in recurrent settings. This paper introduces bipolar activation functions, a simple yet effective architectural trick to keep neural network activations zero-centered by flipping the activation function for every other neuron. This approach stabilizes training and enables the use of much higher learning rates, allowing deep vanilla RNNs and CNNs to perform competitively even without normalization layers.
Methods: The Bipolar Activation Function
The core innovation is elegantly simple: for any standard activation function $f(x)$ (like ReLU or ELU), you define a bipolar version $f_B$ where half of the neurons in a layer behave normally, and the other half are "flipped."
Mathematically, for a neuron index $i$:
- If $i$ is even: $f_B(x_i) = f(x_i)$
- If $i$ is odd: $f_B(x_i) = -f(-x_i)$
For a ReLU, this means half the units are standard ReLUs (passing positive values), and half are "inverted" ReLUs (passing negative values). By flipping half the activations, the positive mean shift of one half cancels out the negative mean shift of the other. This keeps the overall layer mean near zero, a property the authors prove theoretically for Bipolar ReLU.
The authors combine this with two additional techniques to further stabilize deep networks:
- LSUV Initialization: They adapt the Layer-Sequential Unit Variance (LSUV) procedure for RNNs to ensure unit variance at the start of training.
- Skip Connections: To combat "smeared gradients" in very deep RNNs (where the gradient path becomes diagonal through time and layers), they add skip connections every four layers to "straighten" the flow.
Key Insights and Results
The authors evaluated their approach on three standard benchmarks: Penn Treebank (PTB), Text8, and CIFAR-10.
Stabilizing Variance and Enabling Higher Learning Rates
A major bottleneck for aggressive optimization is the mean shift in standard ReLUs. In CNN experiments without Batch Normalization, bipolar activations allowed for learning rates up to 64 times higher than standard activations.
Furthermore, standard ReLU and ELU networks often fail to converge on complex datasets like Text8 due to exploding activations. Their bipolar counterparts train successfully by keeping variance in check.
Competitive Performance
- Penn Treebank: A 36-layer Bipolar ELU (BELU) RNN achieved a test error of 1.270 BPC (Bits Per Character). This outperformed LSTMs using recurrent Batch Normalization (1.32 BPC) and was competitive with other highly regularized LSTM variants.
- Text8: While standard networks failed to converge (DNC), the BELU network reached 1.423 BPC.
- CIFAR-10 (No Batch Norm): When BN was removed from an Oriented Response Network (ORN), standard ReLU had a 9.20% error. Bipolar ReLU slashed this to 4.91%. While still slightly behind the BN-enabled state-of-the-art (2.98%), it represents a massive improvement for BN-free training.
Limitations and Future Work
While bipolar activations significantly close the performance gap, they do not yet fully match networks using Batch Normalization on image tasks. Additionally, for extremely deep RNNs, the "smeared gradient" problem still necessitates the use of skip connections, meaning bipolarity alone isn't a silver bullet for infinite depth.
Future directions could include exploring how bipolarity interacts with other normalization-free techniques or investigating if "learnable" bipolarity (where the network decides which units to flip) could yield even better results.
Summary updated