Interpolation between Residual and Non-Residual Networks

Zonghan Yang, Yang Liu, Chenglong Bao, Zuoqiang Shi

TL;DR

This paper introduces a novel "damped" Ordinary Differential Equation (ODE) model that provides a unified mathematical framework for both Residual Networks (ResNets) and standard non-residual CNNs. By introducing a trainable interpolation parameter, the authors create "In-ResNet" architectures that significantly outperform standard ResNets in robustness against stochastic noise and adversarial attacks, backed by rigorous Lyapunov stability analysis.

What Problem Does This Paper Address?

In recent years, researchers have successfully modeled ResNets as discretizations of Ordinary Differential Equations (ODEs). While this has led to deeper theoretical insights, a significant gap remained: standard, non-residual Convolutional Neural Networks (CNNs) didn't fit into this ODE framework because they lack the identity skip connections that mirror the "state update" in a differential equation.

Furthermore, while ResNets are the industry standard for performance, they are notoriously sensitive to small input perturbations—both random noise and intentional adversarial attacks. This paper seeks to bridge the gap between residual and non-residual architectures and leverage that connection to build more robust deep learning models.

Methods

The authors propose a new damped ODE model: $$\frac{dx(t)}{dt} = -\lambda x(t) + \rho(\lambda)f(x(t), t)$$ Here, $\lambda$ is an interpolation coefficient.

  • When $\lambda = 0$, the model recovers the standard ODE used to describe ResNets.
  • As $\lambda \to \infty$, the model behaves like a non-residual CNN.

By discretizing this damped ODE using the forward Euler method, they derive a new layer structure called In-ResNet. A simplified version of their proposed layer update is: $$x_{n+1} = (1 - \text{ReLU}(\lambda_n))x_n + f_n(x_n)$$ In this architecture, $\lambda_n$ is a trainable parameter for each layer. This allows the network to "learn" the optimal balance between residual and non-residual behavior. Crucially, the authors use Lyapunov stability analysis to prove that this damping term makes the system's equilibrium points more stable, which theoretically translates to better robustness against input perturbations.

Datasets

The models were evaluated on standard image classification benchmarks:

  • CIFAR-10 and CIFAR-100: Used for standard accuracy and optimization difficulty tests.
  • CIFAR-10-C and CIFAR-100-C: Corrupted versions of the datasets containing four types of stochastic noise (impulse, speckle, Gaussian, and shot noise) to test environmental robustness.
  • Adversarial Benchmarks: The models were tested against three common attack methods: Fast Gradient Sign Method (FGSM), Iterated FGSM (IFGSM), and Projected Gradient Descent (PGD).

Key Insights and Ideas

  • Damping as a Robustness Tool: The core "aha" moment is that the damping term $-\lambda x(t)$ weakens the amplitude of the ODE's solution. In neural network terms, this means errors or perturbations introduced at the input are "damped" as they propagate through the layers, rather than being amplified.
  • Learned Interpolation: Instead of choosing between a ResNet or a CNN, the network learns the best architecture per layer. As shown in Figure 1, the learned $\lambda$ values vary across blocks, suggesting that different parts of the network benefit from different levels of "residualness."
Figure 1. Learned interpolation coefficients in In-ResNet-110 and In-ResNet-164 models trained on CIFAR-10 benchmarks.
  • Stability vs. Optimization Trade-off: The authors find that while higher $\lambda$ values (more non-residual behavior) improve robustness, they also make the network harder to optimize, leading to a slight drop in "clean" accuracy.

Key Results

  • Stochastic Noise Robustness: On CIFAR-10-C, the In-ResNet models achieved an average accuracy improvement of over 15% compared to standard ResNet baselines. On CIFAR-100-C, the improvement was over 5%.
  • Adversarial Defense: The proposed models showed significant gains against strong attacks. For example, on CIFAR-10 under PGD attack (radius 2/255), In-ResNet-110 maintained 31.74% accuracy, while the standard ResNet-110 plummeted to just 5.60%.
  • Loss Landscape Smoothing: Visual analysis of the loss landscape (Figure 2) demonstrates that the damping term significantly reduces the volatility of the loss function along the direction of adversarial attacks, making the model less susceptible to gradient-based exploitation.
Figure 2. The input data instance is the 3-th/8-th image in the shot/speckle noise group of the CIFAR-10-C dataset for (a)-(c)/(d)-(f), the ground truth label of which is ship/horse. For (a)-(c)/(d)-(f), ResNet-110 and In-ResNet-110 both make the correct / wrong prediction. (a) and (c) depict the loss landscape of ResNet-110 and In-ResNet-110 along the FGSM attack direction. {(b) and (e)} / {(c) and (f)} illustrate model predictions of {ResNet-110} / {In-ResNet-110} at each grids determined by the FGSM attack direction and a random orthogonal direction.

Limitations and Future Work

The primary limitation identified is the optimization difficulty. As the network moves further away from a pure ResNet (increasing $\lambda$), the vanishing gradient problem becomes more pronounced, which can lead to training failure if the initialization is not carefully tuned.

The authors suggest that future work could involve treating the interpolation coefficients as a "black-box" process or integrating these architectures with advanced data augmentation techniques to further push the boundaries of model robustness.

Summary updated

Abstract

Although ordinary differential equations (ODEs) provide insights for designing network architectures, its relationship with the non-residual convolutional neural networks (CNNs) is still unclear. In this paper, we present a novel ODE model by adding a damping term. It can be shown that the proposed model can recover both a ResNet and a CNN by adjusting an interpolation coefficient. Therefore, the damped ODE model provides a unified framework for the interpretation of residual and non-residual networks. The Lyapunov analysis reveals better stability of the proposed model, and thus yields robustness improvement of the learned networks. Experiments on a number of image classification benchmarks show that the proposed model substantially improves the accuracy of ResNet and ResNeXt over the perturbed inputs from both stochastic noise and adversarial attack methods. Moreover, the loss landscape analysis demonstrates the improved robustness of our method along the attack direction.
BibTeX
@inproceedings{yang2020interpolation,
  author = {Zonghan Yang and Yang Liu and Chenglong Bao and Zuoqiang Shi},
  title = {{Interpolation between Residual and Non-Residual Networks}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
}
Figure