On the Local Hessian in Back-propagation

Huishuai Zhang, Wei Chen, Tie-Yan LiuNeurIPS 2018 · OpenReview

Introduction

Deep learning relies heavily on back-propagation (BP), but BP often requires architectural "tricks" like Batch Normalization (BN) and skip connections to work effectively. While these are commonly attributed to mitigating vanishing gradients, this paper argues that the true key to BP's efficiency lies in the local Hessian—the curvature of the optimization landscape at each individual layer. By reframing BP as a sequence of local matching problems, the authors demonstrate how standard architectural tricks improve this local curvature. Building on these insights, they introduce a novel "Scale-amended SGD" algorithm that outperforms vanilla SGD.

The Back-Matching Framework

To bridge the gap between global optimization and local layer behavior, the authors introduce a framework called back-matching propagation.

Instead of solely focusing on the global loss, they reformulate the network's objective so that each block minimizes a local "matching loss." Essentially, each layer attempts to produce an output that matches a target signal sent down from the layer above. The authors prove a critical proposition: standard back-propagation is mathematically equivalent to taking exactly one gradient descent step on these local matching losses.

Because BP acts as a one-step local solver, its success hinges entirely on the condition of that local problem. If the local Hessian is poorly conditioned (e.g., shaped like a long, narrow valley), a single gradient step will be highly inaccurate, distorting the learning signal as it propagates through the network.

Demystifying Architectural Tricks

The paper provides a unified mathematical explanation for why common deep learning practices work, showing that they fundamentally improve the "spectrum" (eigenvalue distribution) of the local Hessian:

  • Skip Connections: Residual blocks act as preconditioners. Adding a skip connection is mathematically guaranteed to reduce the condition number of the local Gauss-Newton matrix, creating a "rounder," more navigable optimization landscape.
  • Batch Normalization: BN decouples the local Hessian's spectrum from the variance of the weights. This stabilizes the "training pace" across different layers, preventing individual blocks in very deep networks from lagging behind or overshooting.
  • Beyond Vanishing Gradients: The findings reinforce that signal propagation is not just about the magnitude of the gradient, but the quality of the local optimization landscape.

Scale-Amended SGD & Empirical Results

Building on these theoretical insights, the authors propose Scale-amended SGD. Using a scalar approximation of the local Hessian, this algorithm rescales gradients layer-by-layer to balance the training pace across the entire network.

Training and test performance of Scale-amended SGD vs. Vanilla SGD on CIFAR-10 and CIFAR-100.

The authors evaluated their algorithm on CIFAR-10 and CIFAR-100 using a modified VGG13 architecture. As shown in the results, Scale-amended SGD consistently achieved lower training loss and lower test error compared to vanilla SGD. By using local Hessian information to amend the magnitude of updates for each block, the algorithm effectively automates a form of layer-wise learning rate adaptation, leading to a much more stable training trajectory.

Limitations and Future Directions

To make the algorithm computationally tractable, the authors utilized a scalar approximation for the Hessian. While effective, more sophisticated approximations could yield even better results, albeit at a higher computational cost.

Furthermore, the authors suggest a promising avenue for future research: using the local Hessian spectrum as a "fitness" criterion in Neural Architecture Search (NAS). Instead of training thousands of models to completion, researchers could potentially prune the search space early by discarding architectures with poorly conditioned local Hessians.

Summary updated

Abstract

Back-propagation (BP) is the foundation for successfully training deep neural networks. However, BP sometimes has difficulties in propagating a learning signal deep enough effectively, e.g., the vanishing gradient phenomenon. Meanwhile, BP often works well when combining with ``designing tricks'' like orthogonal initialization, batch normalization and skip connection. There is no clear understanding on what is essential to the efficiency of BP. In this paper, we take one step towards clarifying this problem. We view BP as a solution of back-matching propagation which minimizes a sequence of back-matching losses each corresponding to one block of the network. We study the Hessian of the local back-matching loss (local Hessian) and connect it to the efficiency of BP. It turns out that those designing tricks facilitate BP by improving the spectrum of local Hessian. In addition, we can utilize the local Hessian to balance the training pace of each block and design new training algorithms. Based on a scalar approximation of local Hessian, we propose a scale-amended SGD algorithm. We apply it to train neural networks with batch normalization, and achieve favorable results over vanilla SGD. This corroborates the importance of local Hessian from another side.
BibTeX
@inproceedings{zhang2018local,
  author = {Huishuai Zhang and Wei Chen and Tie-Yan Liu},
  title = {{On the Local Hessian in Back-propagation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=SyZIWtbuWr},
}
Figure