On the (Non-)Robustness of Two-Layer Neural Networks in Different Learning Regimes

Elvis Dohmatob, Alberto Bietti

TL;DR

This paper provides a theoretical analysis of adversarial robustness in two-layer neural networks across various learning regimes. It reveals fundamental tradeoffs between prediction accuracy (test error) and robustness, showing that improving one often degrades the other. The study also highlights how random initialization can negatively impact robustness, especially in "lazy" training scenarios.

What Problem Does This Paper Address?

Deep neural networks, despite their success, are notoriously vulnerable to adversarial examples – small, imperceptible input perturbations that can drastically alter predictions. While adversarial training can mitigate this, understanding the underlying causes of this vulnerability is crucial. This paper delves into how factors like random initialization and different learning algorithms (e.g., random features, neural tangent kernel, lazy training) contribute to this fragility in a simplified yet theoretically tractable setting: two-layer neural networks in high dimensions with infinite data. The goal is to disentangle the effects of these factors and uncover inherent tradeoffs between a model's ability to accurately predict and its stability against adversarial attacks.

Methods

The authors employ a theoretical approach, analyzing two-layer neural networks in an asymptotic regime where both the input dimension and network width grow proportionally large. They leverage tools from random matrix theory to derive analytical formulas for test error and robustness. The robustness is quantified using the Dirichlet energy (related to the squared L2 norm of the gradient), which measures the average local stability of the network's predictions. The analysis covers several learning regimes:

  • Fully-trained networks (SGD): Examining the outcome of standard stochastic gradient descent.
  • Random Features (RF): A simplified model where the hidden layer weights are fixed and only the output layer is trained.
  • Neural Tangent (NT) regime: Approximating the network's behavior using the Neural Tangent Kernel, which linearizes the network's dynamics.
  • Lazy training regimes: Investigating scenarios where the network's parameters change minimally during training, often influenced by initialization.

Datasets

The paper does not use specific datasets for training or evaluation. Instead, it relies on a theoretical framework with a synthetic, idealized setting:

  • Input distribution: Standard Gaussian distribution $N(0, I_d)$.
  • Ground truth function: A quadratic function $f^*(x) = x^T Bx + b_0$, where $B$ is a positive semi-definite matrix and $b_0$ is a scalar.
  • Infinite samples: The analysis assumes access to the entire data distribution, sidestepping issues related to finite sample sizes and label noise.

Key Insights and Ideas

  • Inherent Tradeoff: A fundamental tradeoff exists between approximation (test error) and robustness. Improving one metric often leads to a degradation in the other, even in idealized settings with infinite data and no label noise.
  • Initialization Matters: Random initialization can significantly impact a network's robustness. Improperly scaled initial weights can lead to a less robust final model, especially in "lazy" training regimes.
  • Learning Regime Influence: Different learning algorithms (RF, NT, SGD) exhibit distinct robustness-approximation profiles, highlighting the role of the learning dynamics.
  • Task Alignment: The alignment between the target function's structure and the initial weight distribution plays a crucial role in determining both accuracy and robustness.

Key Results

  • Theoretical Tradeoffs: The paper derives analytical formulas for test error and robustness in various learning regimes, explicitly showing the tradeoff between them. For instance, in the SGD regime, the normalized test error and robustness sum to 1 in the limit.
  • Initialization Degradation: For "lazy" training regimes, the paper demonstrates that random initialization can worsen robustness. Specifically, it shows that the robustness of a "lazy" trained network can be lower than that of a vanilla random features model due to inherited nonrobustness from initialization.
  • Empirical Validation: The theoretical predictions are illustrated with numerical experiments, showing a strong agreement between the derived formulas and simulated results.

Limitations and Future Work

  • Two-Layer Networks: The analysis is restricted to two-layer neural networks, which are a simplified model. Extending these findings to deeper architectures remains an open challenge.
  • Infinite Data & High Dimensions: The theoretical framework assumes infinite data and a specific asymptotic regime for high dimensions. While this allows for clean analytical results, it abstracts away complexities present in finite-data, finite-width scenarios.
  • Quadratic Targets: The study focuses on quadratic target functions. Analyzing robustness with more complex target functions could provide broader insights.
  • Future Work: The authors suggest that their work is a first step towards a rigorous analysis of trained neural network robustness. Future research could explore deeper networks, finite-width effects, and different activation functions and data distributions. The paper also raises questions about how to best scale random initialization to enhance robustness.

Summary updated

Abstract

Neural networks are known to be highly sensitive to adversarial examples. These may arise due to different factors, such as random initialization, or spurious correlations in the learning problem. To better understand these factors, we provide a precise study of the adversarial robustness in different scenarios, from initialization to the end of training in different regimes, as well as intermediate scenarios, where initialization still plays a role due to "lazy" training. We consider over-parameterized networks in high dimensions with quadratic targets and infinite samples. Our analysis allows us to identify new tradeoffs between approximation (as measured via test error) and robustness, whereby robustness can only get worse when test error improves, and vice versa. We also show how linearized lazy training regimes can worsen robustness, due to improperly scaled random initialization. Our theoretical results are illustrated with numerical experiments.
BibTeX
@inproceedings{dohmatob2023non,
  author = {Elvis Dohmatob and Alberto Bietti},
  title = {{On the (Non-)Robustness of Two-Layer Neural Networks in Different Learning Regimes}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=guSxooOK9E},
}
Figure