On Connections Between Regularizations for Improving DNN Robustness

Yiwen Guo, Long Chen, Yurong Chen, Changshui Zhang

TL;DR

This paper provides a theoretical bridge between various regularization techniques used to make deep neural networks (DNNs) more robust against adversarial attacks. By analyzing these methods through the lens of piecewise linear models (like ReLU networks), the authors demonstrate that most effective regularizers essentially target the same underlying properties: local cross-Lipschitz constants and prediction confidence.

What Problem Does This Paper Address?

Deep neural networks are notoriously vulnerable to "adversarial examples"—inputs with tiny, intentional perturbations that cause the model to make confident, incorrect predictions. While adversarial training (training on these malicious examples) is effective, it is computationally expensive because it requires generating new attacks at every training step.

As an alternative, researchers have proposed various regularization terms that penalize specific mathematical properties of the network (like the norm of the input gradient or the curvature of the loss surface). However, these methods often seem disconnected, stemming from different rationales. This paper tackles the lack of a unified theoretical understanding, asking: Are these regularizers actually doing the same thing under the hood?

Methods

The authors focus their analysis on DNNs using Rectified Linear Units (ReLU), which are piecewise linear. This mathematical property allows them to derive exact analytic expressions for things that are usually hard to calculate.

Key components of their approach include:

  • Analytic Derivations: They derive the first closed-form expression for the magnitude of an "optimal" adversarial perturbation ($r^*$) in binary classification.
  • Comparative Analysis: They mathematically compare four major regularizers:
  • Jacobian regularization: Penalizes the Frobenius norm of the Jacobian matrix.
  • Input-gradient regularization: Penalizes the $l_2$ norm of the loss gradient.
  • Curvature regularization: Penalizes the spectral norm of the Hessian matrix.
  • Cross-Lipschitz functional: Penalizes the difference between gradients of different class scores.
  • Equivalence Proofs: They prove that under certain conditions (like optimizing only the final layer), Jacobian and cross-Lipschitz regularizations are functionally equivalent.

Datasets

The theoretical findings are validated through experiments on two standard computer vision benchmarks:

  • MNIST: Used for initial tests on simple architectures like LeNet-300-100 and LeNet-5.
  • CIFAR-10: Used to test more complex and deeper models, including VGG-like networks and ResNets.

Key Insights and Ideas

  • The Robustness Formula: The authors show that robustness isn't just about having small gradients. It is a specific interaction between the local cross-Lipschitz constant (how much the difference between class scores changes as you move the input) and the prediction probability (how confident the model is).
  • The Chained Inequality: A major "aha" moment is the discovery of a hierarchy between these methods. They show that: $$\text{Input-gradient} \leq \text{Curvature} \leq \text{Jacobian (Lipschitz)}$$ This explains why these methods often yield similar results but vary in their "strength" and stability.

  • Confidence is the Differentiator: While most regularizers try to reduce the cross-Lipschitz constant, they differ significantly in how they handle model confidence. For instance, input-gradient regularization heavily penalizes low-confidence samples, whereas curvature regularization focuses more on high-confidence ones.

  • Curvature is Often Superior: Empirically, curvature regularization (penalizing the spectral norm of the Hessian) often strikes the best balance, leading to the most promising robustness across different types of attacks (FGSM, PGD, DeepFool, and C&W).

Key Results

  • Theoretical Unification: The paper successfully unifies several disparate lines of research into a single framework based on local network properties.
  • Performance: Experiments confirmed that while all discussed regularizers improve robustness compared to standard training, curvature regularization consistently provided the most stable and effective defense.
  • Equivalence: The authors demonstrated that for single-layer perceptrons or when only the final layer of a DNN is tuned, penalizing the Jacobian is equivalent to penalizing the cross-Lipschitz constant.

Limitations and Future Work

  • ReLU Dependency: Much of the elegant math relies on the piecewise linear nature of ReLU activations. While ReLUs are common, the results might not translate perfectly to networks using smooth activations like Sigmoid or Tanh.
  • Multi-class Complexity: The analysis is most precise for binary classification. In multi-class scenarios, the authors provide lower bounds rather than exact equalities due to the difficulty of decomposing large Hessian matrices.
  • Computational Cost: Curvature regularization, while effective, can be computationally demanding for very large multi-class models, often requiring approximations. The authors suggest that future work should focus on inventing more efficient versions of these principled regularizers.

Summary updated

Abstract

This paper analyzes regularization terms proposed recently for improving the adversarial robustness of deep neural networks (DNNs), from a theoretical point of view. Specifically, we study possible connections between several effective methods, including input-gradient regularization, Jacobian regularization, curvature regularization, and a cross-Lipschitz functional. We investigate them on DNNs with general rectified linear activations, which constitute one of the most prevalent families of models for image classification and a host of other machine learning applications. We shed light on essential ingredients of these regularizations and re-interpret their functionality. Through the lens of our study, more principled and efficient regularizations can possibly be invented in the near future.
BibTeX
@article{guo2021connections,
  author = {Yiwen Guo and Long Chen and Yurong Chen and Changshui Zhang},
  title = {{On Connections Between Regularizations for Improving DNN Robustness}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2021},
  url = {https://openreview.net/forum?id=-Bwedr0uP8l},
}
Figure