TL;DR
Weight Normalization is a simple reparameterization of neural network weights that decouples a weight vector's magnitude from its direction. It provides the training speed-ups of Batch Normalization without introducing messy minibatch dependencies, making it a go-to technique for recurrent networks (LSTMs), generative models, and Reinforcement Learning.
The Problem: Pathological Curvature and Batch Norm Baggage
Training deep neural networks is notoriously difficult because the optimization landscape is often poorly conditioned. The curvature of the objective function can be highly uneven, causing standard first-order gradient descent to stall or oscillate.
While Batch Normalization (BN) revolutionized the field by normalizing hidden unit activations, it comes with significant computational and architectural "baggage." BN calculates statistics (mean and variance) across a minibatch, which introduces stochastic noise into the gradients. This noise is often helpful for regularization in supervised learning but can be disastrous for noise-sensitive applications like Reinforcement Learning (RL) or complex generative models. Furthermore, BN is notoriously difficult to implement in Recurrent Neural Networks (RNNs) because the statistics vary across time steps.
The Method: Weight Normalization
The authors propose a deterministic reparameterization called Weight Normalization. Instead of optimizing a standard weight vector $\mathbf{w}$, they express it as:
$$\mathbf{w} = \frac{g}{\|\mathbf{v}\|} \mathbf{v}$$
Where:
- $\mathbf{v}$ is a $k$-dimensional vector that determines the direction of the weights.
- $g$ is a scalar that determines the magnitude (length) of the weights.
By training the model in terms of $g$ and $\mathbf{v}$ rather than $\mathbf{w}$, the optimization process benefits from two key effects:
- Scaling: The gradient is scaled by $g/\|\mathbf{v}\|$.
- Projection: The gradient is projected away from the current weight vector, which helps "whiten" the gradient and brings it closer to the ideal "natural gradient."
The authors also introduce Data-Dependent Initialization, using a single initial minibatch to set the bias $b$ and scale $g$ so that all activations start with zero mean and unit variance.
Key Insights and Ideas
- Decoupling is Key: Separating the norm from the direction simplifies the optimization landscape. It makes the network much more robust to the choice of learning rate.
- Self-Stabilizing Norm: A fascinating mathematical property of Weight Norm is that the norm of $\mathbf{v}$ tends to grow monotonically during training. As $\|\mathbf{v}\|$ increases, the effective learning rate for the weights decreases, providing a built-in "annealing" effect that stabilizes training.
- Deterministic vs. Stochastic: Unlike Batch Norm, Weight Norm doesn't depend on other examples in the minibatch. This makes it "cleaner" for LSTMs and RL, where the dependency between examples in a batch can lead to instability.
- Mean-only BN: The authors found that while Weight Norm handles the variance of activations well, it doesn't always handle the mean shift. They proposed a "Mean-only Batch Normalization" hybrid that subtracts the minibatch mean but skips the expensive and noisy variance division.
Key Results
The researchers evaluated the method across three distinct domains: Supervised Learning (CIFAR-10), Generative Modeling (MNIST and CIFAR-10 using CVAE and DRAW), and Reinforcement Learning (Atari games using DQN).
Speed & Efficiency in Supervised Learning In CIFAR-10 experiments, Weight Normalization achieved similar or better results than the standard parameterization while being computationally cheaper than Batch Normalization (BN was ~16% slower in their implementation). Combined with Mean-only BN, it achieved a state-of-the-art 7.31% error rate (without data augmentation).
Robustness in Reinforcement Learning In DQN agents playing Atari, Weight Normalization significantly outperformed the standard approach. Batch Normalization actually performed worse than the baseline in this context because the minibatch noise destabilized the Q-learning process.
Faster Convergence in Generative Models For generative models like DRAW and Convolutional VAEs, Weight Normalization reached lower loss levels significantly faster than the standard parameterization.
Limitations and Future Work
The primary limitation is that Weight Normalization alone does not control the "mean" of the activations as effectively as full Batch Normalization. While the authors proposed "Mean-only BN" to fix this, it reintroduces some minibatch dependency.
Additionally, while the data-dependent initialization is powerful, it is difficult to apply to recurrent models (like LSTMs) where the "initial" state of the network is not as clearly defined. Exploring better initialization strategies for recurrent architectures remains an open area for research.
Summary updated