Dynamical Isometry for Residual Networks

Advait Gadhikar, Rebekka BurkholzICLR 2022 · OpenReview

TL;DR

This paper introduces RISOTTO, a novel initialization scheme for ResNets that achieves "dynamical isometry" – a property crucial for stable and fast training of deep neural networks. Unlike previous methods that often struggle without Batch Normalization (BN) or lack feature diversity, RISOTTO balances residual and skip branch contributions, leading to improved performance and stability, even outperforming BN in some cases.

What Problem Does This Paper Address?

Training very deep neural networks, particularly Residual Networks (ResNets), is notoriously sensitive to how the network's weights are initialized. Poor initialization can lead to vanishing or exploding gradients, hindering training speed and generalization. While Batch Normalization (BN) has been a common solution, it introduces computational overhead, breaks sample independence, and can interfere with other training techniques like adversarial training. Existing "BN-free" initialization methods like Fixup and SkipInit aim to mitigate these issues but often suffer from degraded input separability with increasing depth or lack feature diversity. This paper addresses the need for an initialization scheme that provides the benefits of dynamical isometry (DI) for ResNets, enabling stable and effective training without relying on BN, while also maintaining feature diversity.

Methods

The core of the paper is the proposed initialization scheme, RISOTTO (Residual dynamical isometry by initial orthogonality). RISOTTO builds upon the concept of "looks-linear" matrix structures and "Delta Orthogonal" initialization, which aim to preserve signal propagation and maintain the norm of activations.

Key aspects of RISOTTO include:

  • Dynamical Isometry (DI): RISOTTO aims to achieve exact DI, meaning the input-output Jacobian of the initialized residual block has singular values of exactly 1 or -1. This ensures that signal norms are preserved and gradients don't explode or vanish.
  • Balancing Residual and Skip Branches: Unlike methods that heavily favor the skip connection (e.g., setting the residual branch contribution to zero), RISOTTO explicitly balances the contributions of both the residual and skip branches. This is crucial for maintaining feature diversity.
  • Looks-Linear Structure: This technique leverages the property of ReLU activations to separate positive and negative signal components, allowing for orthogonal transformations without losing information.
  • Type-Specific Initialization: RISOTTO provides slightly different initialization strategies for Type B (identity skip connection) and Type C (projection skip connection) residual blocks to account for their structural differences.

Datasets

The paper evaluates RISOTTO on the following standard benchmark datasets:

  • CIFAR10
  • CIFAR100
  • Tiny ImageNet

Key Insights and Ideas

  • DI is Key for ResNets: The paper reinforces the importance of dynamical isometry for stable and effective training of deep ResNets, especially when aiming for BN-free training.
  • Balancing Act is Crucial: Simply suppressing the residual branch (as in some prior methods) can lead to a loss of feature diversity, which is detrimental for generalization. RISOTTO's balanced approach is a significant improvement.
  • RISOTTO Outperforms BN-Free Methods: The proposed scheme demonstrates superior performance compared to existing BN-free initialization methods like Fixup and SkipInit across various datasets.
  • RISOTTO Enhances BN Performance: Surprisingly, RISOTTO can also improve the performance of ResNets when used in conjunction with Batch Normalization, suggesting its benefits extend beyond BN-free scenarios.
  • Understanding BN's Role: The experiments exploring the placement of a single BN layer highlight that its position is critical, and even with optimal placement, RISOTTO can still achieve competitive or superior results.

Key Results

  • Superior BN-Free Performance: RISOTTO consistently outperforms Fixup and SkipInit on CIFAR10, CIFAR100, and Tiny ImageNet when trained without Batch Normalization. For example, on CIFAR100 (50C), RISOTTO achieves 60.63±0.28% accuracy, significantly higher than Fixup (58.25±1.64%) and SkipInit (1±0%).
  • Competitive with BN: In several cases, RISOTTO achieves performance comparable to or even better than standard ResNets trained with Batch Normalization. For instance, on CIFAR100 (50C), RISOTTO with BN achieves 73.11±0.70%, while He Uniform and He Normal achieve 76.17±0.25% and 76.21±0.28% respectively.
  • Stable Training: RISOTTO demonstrates stable training behavior, unlike some other methods that can be susceptible to bad random seeds and lead to failed training runs.
  • Faster Training: Figure 3 suggests that RISOTTO can lead to faster convergence for deep networks like ResNet101.

Limitations and Future Work

  • Theoretical Focus on Fully-Connected: While the paper states that insights transfer to convolutional layers, the theoretical derivations are primarily focused on fully-connected networks for simplicity.
  • Exploration of Other Activations: The current work focuses on ReLU activations. Extending RISOTTO to other activation functions could be a future direction.
  • Hyperparameter Sensitivity: While RISOTTO aims for robustness, further investigation into its sensitivity to hyperparameters like the scaling factor 'a' could be beneficial. The paper does show that a=1 generally yields the best results for Type B blocks.
  • Broader Architecture Exploration: While tested on ResNets, exploring RISOTTO's applicability to other complex architectures could be a valuable next step.

Summary updated

Abstract

The training success, training speed and generalization ability of neural networks rely crucially on the choice of random parameter initialization. It has been shown for multiple architectures that initial dynamical isometry is particularly advantageous. Known initialization schemes for residual blocks, however, miss this property and suffer from degrading separability of different inputs for increasing depth and instability without Batch Normalization or lack feature diversity. We propose a random initialization scheme, RISOTTO, that achieves perfect dynamical isometry for residual networks with ReLU activation functions even for finite depth and width. It balances the contributions of the residual and skip branches unlike other schemes, which initially bias towards the skip connections. In experiments, we demonstrate that in most cases our approach outperforms initialization schemes proposed to make Batch Normalization obsolete, including Fixup and SkipInit, and facilitates stable training. Also in combination with Batch Normalization, we find that RISOTTO often achieves the overall best result.
BibTeX
@inproceedings{gadhikar2022dynamical,
  author = {Advait Gadhikar and Rebekka Burkholz},
  title = {{Dynamical Isometry for Residual Networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=eYOiHguqYO},
}
Figure