Channel Normalization in Convolutional Neural Network avoids Vanishing Gradients

Zhenwei Dai, Reinhard HeckelarXiv 2019 · OpenReview

Abstract

Normalization layers are widely used in deep neural networks to stabilize training. In this paper, we consider the training of convolutional neural networks with gradient descent on a single training example. This optimization problem arises in recent approaches for solving inverse problems such as the deep image prior or the deep decoder. We show that for this setup, channel normalization, which centers and normalizes each channel individually, avoids vanishing gradients, whereas, without normalization, gradients vanish which prevents efficient optimization. This effect prevails in deep single-channel linear convolutional networks, and we show that without channel normalization, gradient descent takes at least exponentially many steps to come close to an optimum. Contrary, with channel normalization, the gradients remain bounded, thus avoiding exploding gradients.

Introduction

Training deep convolutional generators on a single image—a technique central to the Deep Image Prior (DIP) and Deep Decoder—often encounters severe optimization hurdles. These "untrained" networks have revolutionized inverse problems like denoising and MRI reconstruction by fitting to a single target image rather than a large dataset. However, they are notoriously sensitive to architectural choices.

This paper demonstrates that Channel Normalization (CN) is a mathematical necessity for these networks. Without it, gradients vanish exponentially with depth, causing training to stall. We provide both empirical evidence and theoretical proofs to show how CN stabilizes gradients and ensures efficient convergence.

Methodology

The study focuses on Channel Normalization (CN), which standardizes each channel of a convolutional layer individually. For a single training example, CN is functionally equivalent to Instance Normalization or Batch Normalization with a batch size of one.

To analyze its impact, the authors employed a multi-faceted approach:

  1. Empirical Testing: Comparing standard CNNs with and without CN on image-fitting tasks using benchmarks like the Phantom MRI image and ImageNet samples.
  2. Ablation Studies: Testing a "fixed norm" version to determine if the benefit stems from the normalization operation itself or the learnable scale and shift parameters.
  3. Theoretical Analysis: Simplifying the architecture into a Single Channel Linear CNN to analytically prove gradient behavior across many layers.

Theoretical Insights: The Exponential Wall

A core contribution of this work is Theorem 1, which proves that in deep linear CNNs without normalization, the number of gradient descent steps required to reach an optimum grows exponentially with network depth. Under standard initializations, gradients vanish before meaningful updates can occur.

Normalization acts as a "landscape architect," warping the loss surface to create navigable paths. As visualized in Figure 1, the unnormalized loss landscape is a vast, featureless plateau where optimizers stall, whereas the normalized version features a sharp, clear well leading to the solution.

Figure 1: 3D visualizations of the loss surface for multi-channel and single-channel networks, with and without normalization.

Furthermore, the authors found that the trainable scale ($\gamma$) and shift ($\beta$) parameters are secondary; the primary benefit comes from the act of standardization itself, which decouples the length and direction of weight updates and maintains orthogonality between gradients and weights.

Empirical Results

The empirical findings strongly support the theoretical claims. In a 5-layer generator, gradient norms for networks with normalization remained stable (between $10^{-1}$ and $10^{-4}$), while those without normalization plummeted to $10^{-8}$ almost immediately.

Figure 2: Comparison of training loss and gradient norms with and without channel normalization on a phantom image.

As shown in Figure 2, networks with CN reached near-zero training error within a few thousand iterations, whereas unnormalized networks flatlined indefinitely. This stability prevents both vanishing and exploding gradients, ensuring robust training across various depths.

Discussion and Future Work

While this paper provides rigorous proofs for linear networks, empirical evidence suggests these findings extend to non-linear networks using ReLU. However, a full theoretical proof for the non-linear case remains a challenge for future research.

Additionally, while these insights are critical for the "single training example" regime of DIP, future work could explore how channel-wise stability dynamics translate to large-batch training in standard generative models like GANs or Diffusion models. Understanding these fundamental normalization mechanics may lead to more efficient and robust deep generative architectures.

Summary updated

Abstract

Normalization layers are widely used in deep neural networks to stabilize training. In this paper, we consider the training of convolutional neural networks with gradient descent on a single training example. This optimization problem arises in recent approaches for solving inverse problems such as the deep image prior or the deep decoder. We show that for this setup, channel normalization, which centers and normalizes each channel individually, avoids vanishing gradients, whereas, without normalization, gradients vanish which prevents efficient optimization. This effect prevails in deep single-channel linear convolutional networks, and we show that without channel normalization, gradient descent takes at least exponentially many steps to come close to an optimum. Contrary, with channel normalization, the gradients remain bounded, thus avoiding exploding gradients.
BibTeX
@misc{dai2019channel,
  author = {Zhenwei Dai and Reinhard Heckel},
  title = {{Channel Normalization in Convolutional Neural Network avoids Vanishing Gradients}},
  year = {2019},
  eprint = {1907.09539},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.09539},
}

Loading related content...

Figure