Zero Stability Well Predicts Performance of Convolutional Neural Networks

Liangming Chen, Long Jin, Mingsheng ShangAAAI 2022 · DOI

Abstract

Why do some Convolutional Neural Network (CNN) architectures flourish while others collapse? While deep learning often feels like an empirical "black box," researchers are increasingly looking toward the centuries-old field of numerical analysis for answers. This paper demonstrates that zero stability—a property of discrete solvers for differential equations—is a powerful predictor of a CNN's performance, generalization, and robustness. By bridging numerical analysis and deep learning, the authors show that CNNs inspired by Ordinary Differential Equations (ODEs) perform well only if their underlying discrete solvers are "zero-stable." They introduce ZeroSNet, a novel architecture that uses higher-order discretization to achieve superior accuracy and significantly better resistance to input noise compared to standard ResNets.

Introduction

In recent years, researchers have realized that deep residual networks (ResNets) can be viewed as discrete approximations of continuous ODEs. However, simply discretizing an ODE does not guarantee a high-performing model. There has been no clear theoretical framework to explain why certain historical feature combinations (like those in DenseNet or LM-ResNet) work better than others. This research tackles the challenge of architecture design principles by investigating why some configurations lead to better generalization and robustness, identifying "zero stability" as the missing link.

Methods and Theory

The core idea is to treat a CNN as a $d$-th order discretization of an initial value problem. In this view, each layer calculates the next "state" (feature map) based on the current state and several previous "historical" states.

The Root Condition

To determine if a network is "zero-stable," the authors apply the Root Condition from numerical analysis. They analyze a characteristic equation derived from the network's update rule. If the roots of this equation all have a magnitude (modulus) $\le 1$, the network is stable. If any root is $> 1$, the network is unstable—meaning small perturbations in the input will explode as they pass through the layers.

Visualizing feature propagation under different stability conditions: (a) A-stability, (b) BIBO stability, and (c) Zero stability.

ZeroSNet Architecture

The authors propose ZeroSNet, a third-order discretization model. Unlike a standard ResNet, which only considers the immediate previous layer, ZeroSNet blocks take three inputs ($y_n, y_{n-1}, y_{n-2}$) and produce three outputs. The architecture is designed for:

  • Consistency: Mathematically derived coefficients ensure the network accurately tracks the underlying ODE.
  • Stability Region: A flexible parameter, $\lambda$, is used to define the specific range where the network remains zero-stable.
  • Optimal Coefficients: Theoretically calculated weights for historical features maximize stability.

Key Insights

  • Stability = Generalization: Zero stability in deep learning implies that similar inputs lead to similar outputs. A stable network is naturally better at handling the slight differences between training and test samples.
  • The "Performance Cliff": Experiments with PreResNet showed that changing the residual weight $\alpha$ from 1.0 (stable) to 1.5 (unstable) caused CIFAR-10 accuracy to plummet from ~93% to ~87%. This confirms that stability is a requirement for performance, not just a theoretical nicety.
  • Historical Features are Double-Edged: Using information from several previous layers can smooth the loss landscape, but if these features aren't combined using zero-stable coefficients, they make the model more fragile.
  • Noise Suppression: Zero-stable networks act as natural buffers against noise, preventing perturbations from exploding as they propagate through the network.

Experimental Results

The authors evaluated ZeroSNet on CIFAR-10, CIFAR-100, and ImageNet benchmarks:

  • Superior Accuracy: ZeroSNet-Opt consistently outperformed standard ResNets and LM-ResNets. On CIFAR-100, a 110-layer ZeroSNet achieved 74.56% accuracy compared to ResNet's 72.24%.
  • Robustness Gains: When subjected to Gaussian or uniform noise, non-zero-stable models saw accuracy drops of over 12%, while zero-stable ZeroSNets only dropped by approximately 6%.
  • Efficiency: ZeroSNet maintains a parameter count and training time very close to standard PreResNets, offering a performance boost with minimal overhead.

Conclusion

Zero stability provides a rigorous theoretical framework for designing robust and high-performing CNN architectures. While this study focuses on CNNs, the concept of zero stability could potentially be extended to Recurrent Neural Networks (RNNs) or Transformers, where state propagation is also central. Future work will explore higher-order discretizations and the interaction of ZeroSNet with modern training recipes.

Summary updated

BibTeX
@inproceedings{chen2022zero,
  author = {Liangming Chen and Long Jin and Mingsheng Shang},
  title = {{Zero Stability Well Predicts Performance of Convolutional Neural Networks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i6.20576},
}
Figure