Robustness Curves Measured Against the Line

When a new image model is reported as "more robust to corruptions," two things are usually tangled together. The model may simply be better on clean images, and better models also score higher on corrupted ones. Or it may be genuinely less sensitive to noise and blur. The field has two partial remedies that have never been joined. The first is "accuracy on the line," the finding that across thousands of models, accuracy on shifted data is an almost exactly linear function of clean accuracy once both are passed through a probit transform (the inverse of the Gaussian cumulative distribution function). The second is relative corruption metrics such as Relative mCE and the Performance Drop Rate (PDR), which subtract or divide out clean accuracy. Both remedies are computed at a handful of fixed severities, while security evaluation curves (accuracy plotted against perturbation strength) show that the most informative behavior happens between those fixed points.

We propose to fit the accuracy-on-the-line relationship separately at every point along a continuous corruption-severity axis. Each model's robustness then becomes a curve of residuals, meaning how far above or below the line it sits at each severity, instead of a curve of raw or relative accuracy. This gives a precise test of whether relative metrics measure real robustness or are side effects of the line's slope. It also gives a clean account of the ranking inversions reported on security curves. At any fixed severity, a monotone line preserves the ordering of models by clean accuracy. So under the line, every inversion across severity must be a crossing of two residual curves, and inversions can be counted and attributed exactly.

Core Claim

Let $s$ index corruption severity on a continuous perceptual scale, let $z_0$ be a model's probit-transformed clean accuracy (after correcting for chance), and let $z_s$ be its probit accuracy at severity $s$. Using a reference zoo of standard models, we fit

$$ z_s = a(s)\, z_0 + b(s) + \varepsilon_s , $$

where $a(s)$ and $b(s)$ are the slope and intercept of the line at severity $s$, and the residual $\varepsilon_s$ is that model's effective robustness at severity $s$. The effective-robustness curve $\varepsilon(s)$ is the object we propose to study.

We make three claims, and each can fail. First, $a(s)$ and $b(s)$ change smoothly with severity. The goodness of fit $R^2(s)$ stays high up to a corruption-specific threshold and then drops. This is the empirical counterpart of Tripuraneni et al.'s result that the slope of the linear error relationship depends on shift severity. Second, a large part of what Relative mCE and PDR report is predicted by $z_0$ together with $a(s)$ and $b(s)$. Once a model's position on the line is removed, the residual curve is close to uncorrelated with clean accuracy, and the relative metrics are not. Third, most ranking inversions seen on raw security curves occur between models with nearly identical clean accuracy, and these correspond to small residual crossings. The inversions that remain are concentrated in identifiable model families, such as heavily pre-trained or adversarially trained models.

Relation to Prior Work

The motivating evidence comes from three places. Taori et al. defined effective robustness as distance above the standard-model line and evaluated ImageNet-C only at aggregated severities. Miller et al. showed probit linearity with $R^2$ frequently above 0.99, but noted that the fit is weaker for synthetic noise. We extend both by making the line itself a function of severity, which directly asks where Miller et al.'s weakness sets in.

Hendrycks & Dietterich introduced Relative mCE and reported the "robustness plateau": clean error improved from AlexNet to ResNet-50 while relative corruption error barely moved. Xu et al. reported a "fragility paradox" in which Visual Chain-of-Thought (CoT) vision-language models win on absolute accuracy but have average PDR of 26.3% versus 18.6%. Neither paper asks what degradation the line would predict for a model at that clean accuracy. We treat both findings as hypotheses to re-examine through residuals rather than as settled facts.

The method comes from two continuous benchmarks. Shen et al. supply a severity axis based on Visual Information Fidelity, a perceptual measure of how much an image has changed ($\Delta_v \in [0,1]$), and show that ImageNet-C's five levels cover only about 56% of the visual-change range for Gaussian blur. CNS-Bench supplies continuous nuisance shifts generated with a diffusion model and reports ranking changes across scales. Both report raw or human-relative curves, and neither fits a model-zoo line.

On the adversarial side, Dong et al. document inversions between two adversarial training methods, TRADES and PGD adversarial training, across the perturbation budget $\epsilon$. HO-FMN makes full $\epsilon$-curves cheap by finding each input's minimum-norm adversarial perturbation.

Two papers constrain how results can be interpreted. Salaudeen et al. argue that a strong line signals a benchmark whose shift does not penalize shortcuts, the spurious cues a model can rely on. Common corruptions are not designed to reverse shortcuts, so we expect a line at low severity. A drop in $R^2(s)$ at high severity should therefore not be read as the benchmark becoming "well-specified" unless we can separate it from floor effects near chance. Liang et al. show nonlinear, moon-shaped relationships under subpopulation shifts. We therefore also fit a quadratic term in $z_0$ at each severity and report where it becomes necessary.

Research Program

The first component is a model zoo evaluated densely along severity. For ImageNet, we evaluate a zoo of a few hundred publicly available models on the VCR testbed's continuous corruptions (14 types, $\Delta_v$ binned at fine resolution), on CNS-Bench's 14 nuisance shifts, and on ImageNet-C's discrete levels. The zoo covers standard supervised CNNs and transformers, augmentation-trained models, adversarially trained models, and web-scale pre-trained models; Taori et al. evaluated 204 such models. For CIFAR-10, we pair CIFAR-10-C with RobustBench models and HO-FMN minimum-norm curves, which gives an adversarial $\epsilon$-axis. The reference line at each severity is fit only on standard-trained models, following Taori et al., so the interventions of interest cannot shape it. Accuracy is chance-corrected before the probit transform, and bins whose reference accuracy is near chance are flagged rather than fit.

The second component is the analysis. We estimate $a(s)$, $b(s)$, and $R^2(s)$ with bootstrap confidence bands. We locate the severity at which $R^2(s)$ drops below a preset threshold for each corruption. We then decompose every pairwise ranking inversion into a part predicted by the line and a residual-crossing part. Finally, we recompute Relative mCE and PDR under the line: we compute the value each metric would take for a hypothetical model sitting exactly on the line at the same clean accuracy, and report how far each real model deviates from that value.

The third component is an application to multimodal models. We repeat the fit on VLM-RobustBench across as many open vision-language models as are available, and ask whether Visual CoT models fall below the line. If they do, the fragility paradox is real. If their larger drop is what their clean accuracy predicts, it is an artifact of the metric. VLM-RobustBench covers 49 augmentation types.

Baselines

The naive reference point is the raw security curve: accuracy against severity, summarized by its area, which is the expected-robustness measure of Ngnawe et al.. It is computed on exactly the same images and models. It is the default in the curve literature, and we expect it to track clean accuracy closely. Our residual summary beats it if it has near-zero rank correlation with clean accuracy across the zoo while still separating the model families known to leave the line, such as web-scale pre-trained models.

The strongest existing relative metric is Relative mCE from Hendrycks & Dietterich, computed on ImageNet-C with the AlexNet normalization. The reported reference values are an mCE of 76.7 for ResNet-50 against 100 for AlexNet; the file gives no Relative mCE figure for ResNet-50. PDR from Xu et al. is the self-normalized alternative. For the VLM comparison, the reported figure is average PDR of 26.3% for Visual CoT models versus 18.6% for standard VLMs. Both metrics use the same model outputs as our method and need no tuning, so the comparison is fair by construction. We beat them if a substantial, bootstrap-significant fraction of their variance across the zoo is explained by clean accuracy through the fitted line. We also require that our residual curves predict each model's held-out-severity accuracy better than the relative metrics do.

The most direct competitor is static effective robustness from Taori et al.: a single line fit on severity-averaged ImageNet-C. It isolates the ingredient we add, namely severity dependence. The first ablation fits the line only at ImageNet-C's five discrete levels and interpolates $a(s)$ and $b(s)$ between them. A second ablation drops the quadratic term. Both are compared on held-out dense bins of the VCR axis, using the same zoo and the same number of fitted parameters per severity.

Why not just…?

Why not just use static effective robustness on aggregate ImageNet-C? It already removes the clean-accuracy confound. However, it assumes that a model's advantage is the same at every severity. That assumption is exactly what raw-curve inversions in Dong et al. and CNS-Bench call into question. If the residual curves turn out nearly flat, with low-severity residual predicting high-severity residual almost perfectly, then static effective robustness is sufficient and our severity-indexed version adds nothing. We will report that outcome as a negative result.

Why not just fit the line at ImageNet-C's five levels? This is the most serious threat to the proposal, because it recovers most of the idea at no extra cost. The case for the continuous axis rests on Shen et al.'s finding that the five levels miss much of the perceptual range. That finding is about raw accuracy, not about the line. The discrete-interpolation ablation settles the question: if it predicts dense-bin accuracy as well as the full fit, the practical recommendation becomes "fit per level," and the continuous data serve only to validate that recommendation.

Why not just keep using Relative mCE or PDR, which were designed to remove clean accuracy? Subtracting or dividing by clean accuracy removes the confound only if the line has unit slope and zero intercept in the space where the subtraction happens. Miller et al.'s probit fits make no such guarantee. Our decomposition measures how much of each relative score is implied by the line. If that share is small, the relative metrics are vindicated.

Decisive Checks

The proposal succeeds on three checks. First, $R^2(s)$ shows a reproducible, corruption-specific breakdown severity that is stable across bootstrap resamples of the zoo and survives chance correction. Second, over half of raw-curve ranking inversions on CNS-Bench and VCR fall between models whose clean accuracies differ by less than the line's prediction error. Third, the Visual CoT PDR gap either shrinks or persists once it is measured against the line, which gives a direct answer to whether the fragility paradox is real. On the CIFAR-10 adversarial axis, the file reports only a single point, TRADES at about 56.8% under the BIM attack at $\epsilon = 8/255$ (Dong et al.). There is no reported result on Lacuna for a zoo-wide line fit against $\epsilon$. We therefore treat whether the line exists at all under adversarial attack as an open question, and expect it to break at small $\epsilon$.