Pushing Stochastic Gradient towards Second-Order Methods -- Backpropagation Learning with Transformations in Nonlinearities

Tommi Vatanen, Tapani Raiko, Harri Valpola, Yann LeCun

TL;DR

This paper introduces LTMLP (Linear Transformations in Multi-Layer Perceptrons), a method that transforms hidden neuron activations to have zero mean, zero slope, and unit variance. These simple linear transformations make standard first-order backpropagation behave like computationally expensive second-order optimization methods, leading to significantly faster convergence and better performance on deep networks.

What Problem Does This Paper Address?

Training deep neural networks is notoriously difficult because the "loss landscape" is often warped and highly non-convex. Standard Stochastic Gradient Descent (SGD) follows the steepest slope, but in deep models, this direction rarely points directly toward the true minimum.

To fix this, researchers often look to second-order methods (like Newton’s method or Natural Gradient), which use information about the curvature of the landscape—captured by the Hessian or Fisher Information Matrix—to take more direct paths. However, calculating and inverting these massive matrices is computationally ruinous for modern deep networks. The authors ask a fundamental question: Can we change the model's architecture so that a simple first-order gradient naturally looks like a second-order update?

Methods

The authors propose three specific transformations for the nonlinear activation functions (like tanh) in a Multi-Layer Perceptron (MLP):

  1. Zero Mean: The outputs of hidden neurons are centered to average to zero.
  2. Zero Slope: The average derivative (slope) of the activation function is forced to zero.
  3. Unit Variance (The Novelty): The scale of the outputs is normalized.

To ensure the model doesn't lose expressive power when the nonlinearities are constrained in this way, the authors add linear shortcut connections that bypass the nonlinear layers. These shortcuts handle the linear dependencies, while the transformed neurons focus purely on the nonlinear "residuals."

Mathematically, they show that these transformations push the Fisher Information Matrix closer to an identity matrix. When this happens, the standard gradient and the natural gradient (a second-order method) become nearly identical, allowing for much faster learning without the heavy mathematical overhead.

Key Insights and Ideas

  • Geometry Matters: By centering and scaling activations, the authors are essentially "sphericalizing" the loss surface. This makes the steepest descent direction much more accurate.
  • Reducing Competition: In standard networks, the bias term, linear components, and nonlinear components often "compete" to explain the same data. By forcing nonlinearities to have zero mean and zero slope, the authors decouple these components. The linear shortcuts do the heavy lifting for simple patterns, while neurons focus on complex ones.
  • The Natural Gradient Shortcut: The most elegant insight is that you don't need to calculate a Hessian if you can design a network where the Hessian is approximately the identity matrix.
  • The "Dead Neuron" Trap: They discovered that the third transformation (unit variance) is a double-edged sword. While it speeds up learning, it can occasionally push neurons into a state where both inputs and outputs are near zero, leading to poor local optima.

Key Results

  • Massive Step Sizes: Because the gradients were more "stable" and accurate, the LTMLP could use a learning rate (1.2) nearly three times larger than a regular MLP (0.45).
  • Superior Accuracy: On the MNIST dataset, the transformed networks achieved as few as 112 errors (out of 10,000), significantly outperforming regular backpropagation and rivaling state-of-the-art techniques of the time.
  • Theoretical Alignment: Using finite difference approximations of the Hessian, the authors proved that the angle between the SGD gradient and the "ideal" second-order update was much smaller in their transformed networks than in standard ones.
Eigenvalues and gradient angles for LTMLP vs. regular networks

As shown above, the LTMLP architecture successfully flattens the eigenvalue spectrum of the Fisher Information Matrix (left) and drastically reduces the angle between the standard gradient and the optimal second-order update (right), confirming the theoretical motivations of the paper.

Limitations and Future Work

  • Autoencoder Instability: The unit variance transformation caused issues in the decoder layers of deep bottleneck autoencoders. The "dead neuron" problem meant that the autoencoder would sometimes diverge or converge to a poor reconstruction.
  • Hyperparameter Nuisance: While the method reduces the need for complex optimizers, it introduces its own constraints (like capping the scaling factor) that require careful tuning.
  • Integration with Modern Tech: The authors suggest that future work should explore combining these transformations with regularization techniques like Dropout to further stabilize the learning process.

Summary updated

Abstract

Recently, we proposed to transform the outputs of each hidden neuron in a multi-layer perceptron network to have zero output and zero slope on average, and use separate shortcut connections to model the linear dependencies instead. We continue the work by firstly introducing a third transformation to normalize the scale of the outputs of each hidden neuron, and secondly by analyzing the connections to second order optimization methods. We show that the transformations make a simple stochastic gradient behave closer to second-order optimization methods and thus speed up learning. This is shown both in theory and with experiments. The experiments on the third transformation show that while it further increases the speed of learning, it can also hurt performance by converging to a worse local optimum, where both the inputs and outputs of many hidden neurons are close to zero.
BibTeX
@inproceedings{vatanen2013pushing,
  author = {Tommi Vatanen and Tapani Raiko and Harri Valpola and Yann LeCun},
  title = {{Pushing Stochastic Gradient towards Second-Order Methods -- Backpropagation Learning with Transformations in Nonlinearities}},
  booktitle = {ICONIP},
  year = {2013},
  doi = {10.1007/978-3-642-42054-2\_55},
}
Figure