Abstract
We present weight normalization: a reparameterization of the weight vectors in a neural network that decouples the length of those weight vectors from their direction. By reparameterizing the weights in this way we improve the conditioning of the optimization problem and we speed up convergence of stochastic gradient descent. Our reparameterization is inspired by batch normalization but does not introduce any dependencies between the examples in a minibatch. This means that our method can also be applied successfully to recurrent models such as LSTMs and to noise-sensitive applications such as deep reinforcement learning or generative models, for which batch normalization is less well suited. Although our method is much simpler, it still provides much of the speed-up of full batch normalization. In addition, the computational overhead of our method is lower, permitting more optimization steps to be taken in the same amount of time. We demonstrate the usefulness of our method on applications in supervised image recognition, generative modelling, and deep reinforcement learning.
Introduction
Training deep neural networks is notoriously difficult because the optimization landscape is often poorly conditioned. If the Hessian matrix has a high condition number, first-order gradient descent methods struggle to make progress, a phenomenon known as "pathological curvature."
While Batch Normalization (BN) was a breakthrough in solving this by normalizing neuron activations, it has significant drawbacks. BN introduces dependencies between examples in a minibatch, adding stochastic noise to the gradients. While often helpful for regularization in image classification, this noise can be disastrous for noise-sensitive tasks like Deep Reinforcement Learning or Generative Modeling. Furthermore, BN is difficult to implement in Recurrent Neural Networks (RNNs) and LSTMs because statistics vary across time steps.
Methods: Weight Normalization
The authors propose a deterministic reparameterization called Weight Normalization. Instead of parameterizing a neuron with a standard weight vector $\mathbf{w}$, they express it as:
$$\mathbf{w} = \frac{g}{||\mathbf{v}||} \mathbf{v}$$
Here, $\mathbf{v}$ is a $k$-dimensional vector that determines the direction of the weights, and $g$ is a scalar that determines the magnitude (the Euclidean norm).
By training $g$ and $\mathbf{v}$ directly using stochastic gradient descent, the authors achieve two things:
- Improved Conditioning: Decoupling the norm from the direction makes the optimization problem easier for the gradient descent algorithm to navigate.
- Self-Stabilizing Gradients: As $|\mathbf{v}||$ grows during training, the effective learning rate for the weights decreases, providing a built-in stabilization mechanism.
The authors also introduce Data-Dependent Initialization, where they use a single initial minibatch to set the bias $b$ and scale $g$ so that all neurons start with zero mean and unit variance.
Experimental Results
Supervised Learning (CIFAR-10)
On the CIFAR-10 dataset, Weight Normalization significantly accelerated convergence compared to standard parameterization. WN combined with "mean-only" Batch Normalization achieved a 7.31% test error, outperforming standard BN (8.05%) and setting a state-of-the-art result for models without data augmentation at the time.
Generative Modeling
In generative tasks, WN proved particularly effective for recurrent architectures like DRAW. Unlike BN, WN's deterministic nature avoids the problematic noise that can hinder the optimization of marginal log-likelihood bounds. In the DRAW model, WN led to significantly faster convergence, reaching lower (better) bounds much earlier than standard parameterization.
Reinforcement Learning
On Atari games, DQN with Weight Normalization consistently outperformed the standard version. For example, on Space Invaders, the WN-equipped agent achieved a score of 2,179 compared to the standard agent's 1,779.
Discussion and Key Insights
The primary insight of this work is that normalizing the weights themselves can provide the benefits of activation normalization without the overhead or minibatch dependencies of BN.
- Robustness: WN is much more robust to the choice of learning rate due to its self-stabilizing property.
- Efficiency: WN has lower computational overhead than BN, resulting in roughly 16% faster training per step in the authors' implementations.
- Limitations: While WN fixes the scale of weights, it does not fix the mean of activations. For some architectures, combining WN with a simplified "mean-only" version of Batch Normalization is necessary for optimal results. Additionally, data-dependent initialization remains challenging to apply to complex temporal models like RNNs.
Summary updated