Image classifiers that score well on clean test sets often fail on blurred, noisy, foggy or compressed versions of the same images. Benchmarks such as CIFAR-10-C and ImageNet-C measure this by applying fixed corruptions at five severity levels. A separate body of theory studies the "lazy" regime of training. In that regime a very wide or large-initialization network barely moves its weights, behaves like a fixed kernel method (a linear model on random features frozen at initialization), and so never learns task-specific features. The obvious bridge is to ask whether lazy networks are less robust to corruptions. We propose a sharper version. In the lazy regime, a network's error under each corruption can be forecast before training from its initial kernel. The extra robustness that feature learning buys is then the gap between that forecast and the trained network. We predict this gap differs by corruption type: it should be large where the corruption adds energy along directions the label does not depend on (noise, pixelation, JPEG artifacts), and near zero, or negative, where the corruption erases the structure the label does depend on (fog, contrast, defocus).
If the prediction holds, it turns "the robustness plateau" from an empirical shrug into an explained effect. It would tell practitioners which corruption failures a change of parameterization can address, and which ones only data or architecture can address. If it fails, and every corruption improves only in proportion to clean accuracy, that is an equally useful negative result. It would mean the lazy-versus-rich distinction is irrelevant for average-case corruptions, unlike for adversarial ones.
Core Claim
The claim has three parts. First, along the continuum from rich to lazy training, per-corruption error of near-lazy networks is predicted by kernel regression with the initialization tangent kernel, which we call the Neural Tangent Kernel (NTK). Second, at matched clean accuracy, moving toward the rich regime reduces error mainly on corruption families whose displacement falls in the low-eigenvalue "tail" of that initial kernel. Third, this tail energy, computed per corruption from the initial kernel alone, predicts the size of the feature-learning benefit.
The mechanism combines two prior results. Regularization Matters proves that kernel methods need $\Omega(d^2)$ samples to ignore $d-2$ irrelevant noise coordinates, while an actively trained two-layer net needs $O(d)$, because it learns to select the relevant coordinates. An input corruption that perturbs irrelevant directions should therefore hurt a kernel predictor, which spreads weight over them, and spare a feature-learning one. What Can the Neural Tangent Kernel Tell Us About Adversarial Robustness? shows that NTK tail eigenvectors are high-frequency and non-robust while top eigenvectors are shape-like and robust. Together these motivate our split by corruption family.
The proposal is also constrained by a third result. Adversarial Robustness is at Odds with Lazy Training proves lazy two-layer nets have input-gradient norm $\Omega(\sqrt{d})$, which makes worst-case perturbations of size $O(1/\sqrt{d})$ sufficient to flip predictions. For isotropic Gaussian noise $\delta \sim \mathcal{N}(0,\sigma^2 I)$ with per-pixel scale $\sigma$, a first-order expansion gives
$$\mathbb{E}\big[(f(x+\delta)-f(x))^2\big] \approx \sigma^2\,\|\nabla_x f(x)\|^2 ,$$
where $f$ is the network output. Combined with their bound, the root-mean-square output change grows like $\sigma\sqrt{d}$. That is a factor $\sqrt{d}$ milder than the adversarial case, but it still grows with resolution at fixed severity. This gives a quantitative, falsifiable scaling prediction for the noise family of a corruption benchmark. Their paper does not test this case, since it studies only gradient-aligned attacks on binary MNIST. Blur and fog, which are not random and are low-frequency, are not covered by this argument, and we expect them to behave differently.
Research Program
The central construction is a controlled laziness dial. Following the setup of the phase diagram of Geiger, Petrini and Wyart, we train $f_\alpha(x) = \alpha\,(h(w,x) - h(w_0,x))$. Here $h$ is the network with weights $w$, $w_0$ is the initialization, and $\alpha$ is a scale, with the learning rate scaled as $1/\alpha^2$. Large $\alpha$ forces laziness and small $\alpha$ forces feature learning, while architecture, data, and optimizer are held fixed. The Scaling Path proves this interpolation is continuous for two-layer nets, so we expect smooth trends rather than a phase jump. That justifies regressing robustness on $\log\alpha$.
A second, practically relevant dial is width under standard parameterization versus Maximal Update Parametrization (μP). μP is the scaling from Tensor Programs V under which every layer's features keep moving by a constant amount as width grows. Standard parameterization drifts toward laziness as width grows.
The lazy forecast for each corruption $c$ is ordinary kernel regression:
$$\hat f_c(x) = K_0(T_c(x), X)\,K_0(X,X)^{-1} Y .$$
Here $K_0$ is the empirical NTK at initialization, $T_c$ applies corruption $c$, and $X, Y$ are training inputs and labels. The per-corruption feature-learning residual is the trained network's corrupted error minus the forecast's corrupted error, measured at matched clean accuracy. The spectral predictor is the fraction of the corruption displacement's kernel energy that falls outside the top-$k$ eigenvectors of $K_0$.
Because NTK computation is quadratic in dataset size, the file notes that such analyses are restricted to small models and CIFAR-scale data. We therefore compute kernels on CIFAR-10 training subsets of a few thousand to ten thousand images, and validate that conclusions are stable across subset size.
We also measure the internal "feature collapse" described by Feature Collapse Under Corruption. That paper found corruptions push deep-layer representations to high entropy and treated this as something to fine-tune away. We ask whether collapse grows with $\alpha$. This connects their empirical finding to the claim in Global Convergence and Rich Feature Learning under μP that μP features keep non-degenerate covariance during training.
Baselines
The naive reference point is a standard-parameterization network at default width and initialization, trained normally on clean CIFAR-10 and evaluated on CIFAR-10-C. It anchors what "doing nothing about regime" gives. All other settings share its data, epochs, augmentation, and a fixed per-setting learning-rate search budget, so differences cannot come from extra tuning.
The lazy extreme is the analytic kernel forecast itself, which requires no network training. It is the right competitor for the claim that near-lazy networks are predictable from initialization. The forecast succeeds if its per-condition error ranks match large-$\alpha$ networks across all 75 corruption-severity conditions.
The strongest competitor for explaining corruption error is not a method from the file but an accuracy-on-the-line rule: predict each model's corrupted accuracy from its clean accuracy by a per-corruption linear fit across checkpoints and settings. If this rule explains the regime effect as well as our residual does, our claim is empty. Beating it means the regime predicts corrupted accuracy significantly beyond what clean accuracy predicts, with the sign differing by corruption family. We fit the rule on the same pool of runs, using held-out runs for evaluation.
As an intervention reference we report Dem-HEC from Feature Collapse Under Corruption. It reports RepVGG-A0 accuracy under severe Gaussian noise on CIFAR-10-C rising from 14.37% to 69.29% after fine-tuning. We also report Gaussian-noise data augmentation, and the ImageNet-C reference that a standard ResNet-50 reaches mCE 76.7 (Hendrycks and Dietterich). We do not expect a regime change alone to match a corruption-targeted fine-tune. The test is whether the regime effect is additive with augmentation or redundant with it.
The key ablation isolates feature learning from kernel quality. We rerun the forecast with the final empirical kernel (the "after kernel") in place of $K_0$. If the after-kernel forecast closes the residual, the benefit lives entirely in how features changed, not in the readout.
Why not just…?
Why not just train with corruption-style augmentation? It works for the corruptions it covers. However, Hendrycks and Dietterich warn that training on benchmark corruptions inflates scores without improving the underlying representation. Augmentation also answers a different question, since it changes the data rather than explaining why some models degrade gracefully. We include noise augmentation crossed with $\alpha$. If augmentation erases the regime effect on noise but not on fog, that is itself evidence for our directional account.
Why not just attribute everything to clean accuracy? This is the real threat, because richer training usually improves clean accuracy, and corrupted accuracy tracks clean accuracy. We settle it two ways. First, we compare models at matched clean error, using early-stopped checkpoints of large-$\alpha$ runs against small-$\alpha$ runs. Second, we use the clean-prediction-as-label agreement metric from Dooley et al. alongside relative corruption error. If residuals vanish under matching, we report that outcome.
Why not just make standard-parameterization networks wider, or use adversarial training? Under standard parameterization, width pushes toward laziness. CompleteP for RL found this slows feature evolution under distribution shift, and μP-style scaling gave 33% higher reward there. Our width sweep tests whether the same holds for image corruptions. Adversarial training, per Tsilivis and Kempe, concentrates the kernel onto its top eigenvalues. Our account then predicts it should help noise-family corruptions and do little for fog or contrast, and we test that directly with one adversarially trained run per width.
Proposed Experiments
The first experiment is a controlled toy case. We use the coordinate-selection distribution of Regularization Matters and inject "corruptions" of two kinds: Gaussian noise on the $d-2$ irrelevant coordinates, or noise on the two relevant ones. We predict the kernel predictor degrades under the first while rich networks are unaffected, and that both degrade equally under the second. Failure here would falsify the mechanism cheaply.
The second experiment is the main study. We train small CNNs and narrow ResNets on CIFAR-10 across roughly six values of $\alpha$, and across a width sweep in both standard parameterization and μP. We evaluate on all 15 CIFAR-10-C corruptions at all 5 severities, recording decay curves, relative corruption error, prediction agreement with clean outputs, and layer entropy. Tiny-ImageNet-C repeats the width sweep at higher resolution. A resolution sweep, built by downsampling CIFAR-10 and recomputing the corruptions at fixed per-pixel severity, tests the $\sigma\sqrt{d}$ noise prediction for large-$\alpha$ models. We expect that scaling to be visibly weaker at small $\alpha$.
Three outcomes are decisive. The first is whether the initial-kernel forecast ranks per-condition errors of large-$\alpha$ networks with high rank correlation. The second is whether, at matched clean accuracy, the feature-learning residual is significantly larger on the noise and digital families than on the weather and blur families. The third is whether the spectral tail-energy score of each corruption predicts its residual better than the accuracy-on-the-line fit does. A uniform, family-independent residual would refute the central claim. A residual that favors laziness on low-frequency corruptions would reveal a real cost of feature learning.
References
- Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel
- What Can the Neural Tangent Kernel Tell Us About Adversarial Robustness?
- Adversarial Robustness is at Odds with Lazy Training
- Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Training
- On the Effect of Initialization: The Scaling Path of 2-Layer Neural Networks
- Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
- Feature Collapse Under Corruption: An Entropy Perspective on Robust Neural Networks
- Global Convergence and Rich Feature Learning in $L$-Layer Infinite-Width Neural Networks under $\mu$ Parametrization
- Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
- Robustness Disparities in Face Detection
- CompleteP for RL: Maintaining Feature Learning When Scaling Deep Reinforcement Learning