On the (Non-)Robustness of Two-Layer Neural Networks in Different Learning Regimes

Elvis Dohmatob, Alberto Bietti

TL;DR

Even with infinite training data and no label noise, two-layer neural networks face a fundamental, mathematically exact tradeoff: improving test accuracy directly worsens adversarial robustness. Across multiple training regimes—including SGD, Random Features, and Neural Tangent kernels—the sum of normalized test error and normalized robustness error stubbornly equals one.

What Problem Does This Paper Address?

Adversarial vulnerability in deep learning is often blamed on finite sample sizes, label noise, or spurious correlations. Prior theoretical work suggested that overparameterization might resolve this issue in noiseless settings. This paper investigates whether adversarial fragility persists under ideal conditions: an infinite-sample regression problem with zero label noise, mapping standard Gaussian inputs to a structured quadratic teacher model ($f_\star(x) = x^\top B x - \text{tr}(B)$). By analyzing two-layer student networks across different optimization regimes, the authors seek to isolate how network inductive bias, feature learning, and initialization scale inherently dictate adversarial vulnerability.

Methods

The paper examines high-dimensional proportional asymptotics where input dimension $d \to \infty$ and network width $m \to \infty$ such that the parameterization ratio $m/d \to \rho \in (0, \infty)$.

  • Robustness Metric: Robustness error is quantified via the expected squared gradient norm (Dirichlet energy w.r.t. Gaussian input distribution $P_x$): $\varepsilon_{\text{rob}}(f) := \mathbb{E}_{P_x}[\|\nabla_x f(x)\|^2]$. This is proven to lower-bound the local Lipschitz constant and tightly capture average-case sensitivity to local and universal adversarial perturbations.
  • Studied Regimes:
  • Full SGD: Two-layer networks with quadratic activations trained on population risk where the first layer learns low-rank approximations of the teacher matrix $B$.
  • Random Features (RF): Random first-layer weights ($w_j \sim \mathcal{N}(0, \Gamma)$), training only the linear output layer.
  • Neural Tangent (NT): First-order Taylor linearization of the network around initialization, ignoring the initial function offset.
  • Lazy Training (NTL / RFL): Linearized training retaining the non-zero initial function offset ($f_{\text{init}}$).
  • Mathematical Tools: Random matrix theory (Silverstein's fixed-point equations, resolvent analysis) and Hermite polynomial expansions of non-linear activations.

Datasets

  • Synthetic Gaussian Data: The paper strictly uses simulated data where inputs $x \sim \mathcal{N}(0, I_d)$ in dimension $d = 450$ (with $n = 10^6$ samples acting as an empirical proxy for infinite data in experiments). No real-world benchmark datasets (like ImageNet or CIFAR) are used, as the work is purely theoretical and validated on controlled simulations.

Key Insights and Ideas

  • The Conservation Tradeoff ($\widetilde{\varepsilon}_{\text{test}} + \widetilde{\varepsilon}_{\text{rob}} = 1$): Across SGD, RF, and NT regimes, minimizing test error forces the student model to increase its input gradients to match the curvature of the teacher model. In the under-parameterized or misaligned regimes, the student network is actually more robust (less sensitive to input perturbations) than the ground truth teacher, but this stability directly incurs prediction error.

  • Task Alignment Dictates the Tradeoff in RF: In the random features regime, test performance and robustness depend directly on the spectral alignment $\alpha = \text{tr}(B\Gamma) / (\|B\|_F \|\Gamma\|_F)$. As $\rho \to \infty$, normalized test error approaches $1 - \alpha^2$ while normalized robustness error approaches $\alpha^2$.

  • The Penalty of "Lazy" Initialization: In lazy training (NTL), retaining the random initialization term $f_{\text{init}}$ does not affect asymptotic test error, but it strictly degrades robustness by an additive penalty: $$\mathbb{E}[\widetilde{\varepsilon}_{\text{rob}}(f_{\text{NTL}})] = \mathbb{E}[\widetilde{\varepsilon}_{\text{rob}}(f_{\text{NT}})] + \mathbb{E}[\widetilde{\varepsilon}_{\text{rob}}(f_{\text{init}})] + o_d(1)$$ If initialization variance is improperly scaled (large relative to the true signal $B$), the final model inherits the fragility of a purely random network.

  • General Poincaré Lower Bound: Using the Poincaré inequality for Gaussian measures, the authors show that for any smooth model $f$, $\sqrt{\varepsilon_{\text{test}}(f)} + \sqrt{\varepsilon_{\text{rob}}(f)} \ge \|f_\star\|_{L^2(P_x)}$, proving that test error and gradient sensitivity are fundamentally linked by data geometry.

Key Results

  • SGD Regime: $\widetilde{\varepsilon}_{\text{rob}}(f_{\text{SGD}}) \to \frac{\|B\|_{F, m}^2}{\|B\|_F^2} \le 1$. If width $m \ge \text{rank}(B)$, the network achieves perfect robustness matching the teacher ($\widetilde{\varepsilon}_{\text{rob}} \to 1$) and zero test error ($\widetilde{\varepsilon}_{\text{test}} \to 0$). If $m < \text{rank}(B)$, $\widetilde{\varepsilon}_{\text{test}} + \widetilde{\varepsilon}_{\text{rob}} \to 1$.
  • RF Regime: For general activations (including ReLU), $\widetilde{\varepsilon}_{\text{rob}}(f_{\text{RF}}) = \frac{\tau^2 (2\kappa \psi_1^2 + \psi_2)}{\|B\|_F^2 (2\kappa \psi_1 + 2)^2} + o_{d, \mathbb{P}}(1)$.
  • Empirical Match: Numerical simulations ($d=450$, width $m \in [10, 1500]$, quadratic and ReLU activations) match asymptotic predictions nearly perfectly (Figures 1 and 2).

Limitations and Future Work

  • Model Architecture: The exact equality tradeoffs are derived specifically for two-layer shallow networks and structured quadratic teacher targets; deep multilayer networks are not theoretically characterized.
  • Data Distribution: The exact results heavily rely on Gaussian inputs ($x \sim \mathcal{N}(0, I_d)$) to leverage rotational invariance and Hermite expansions.
  • Sample Limits: The paper focuses on $n = \infty$. While Proposition C.1 shows a similar dynamic for finite-sample linear regression, extending the nonlinear RF/NT matrix concentration results to finite $n$ remains an open challenge.

Summary updated

Abstract

Neural networks are known to be highly sensitive to adversarial examples. These may arise due to different factors, such as random initialization, or spurious correlations in the learning problem. To better understand these factors, we provide a precise study of the adversarial robustness in different scenarios, from initialization to the end of training in different regimes, as well as intermediate scenarios where initialization still plays a role due to “lazy” training. We consider over-parameterized networks in high dimensions with quadratic targets and infinite samples. Our analysis allows us to identify new tradeoffs between approximation (as measured via test error) and robustness, whereby robustness can only get worse when test error improves, and vice versa. We also show how linearized lazy training regimes can worsen robustness, due to improperly scaled random initialization. Our theoretical results are illustrated with numerical experiments.
BibTeX
@inproceedings{dohmatob2023non,
  author = {Elvis Dohmatob and Alberto Bietti},
  title = {{On the (Non-)Robustness of Two-Layer Neural Networks in Different Learning Regimes}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=guSxooOK9E},
}

Loading related content...

Figure