Is the Clean-Accuracy Tax of Noise Robustness a Noise-Model Mismatch?

Classifiers that are trained to tolerate noisy inputs usually get worse on clean inputs. This loss of accuracy on unperturbed data is often treated as an unavoidable price of robustness. Most of that training adds the same Gaussian noise to every pixel of a finished, display-ready image. Real cameras and detectors do not produce noise that way. They count photons. Their noise grows with brightness: the variance at a pixel is roughly $\sigma^2(x) = \alpha x + \sigma_g^2$, where $x$ is the true linear intensity, $\alpha$ is the sensor gain that scales photon (shot) noise, and $\sigma_g^2$ is the brightness-independent electronic read noise. A dim pixel and a bright pixel therefore carry very different amounts of uncertainty. A uniform noise model asks the classifier to survive corruptions in dark regions that no camera could ever produce.

This note proposes a specific test with a specific prediction. If noisy training views are generated as the same scene photographed at lower exposure, the label is invariant by construction and the noisy views stay on the manifold of real images. To do this, finished images are mapped back to linear sensor values, photon-counting noise is applied at a randomly reduced exposure, and the image is re-rendered. Under this training, most of the clean-accuracy tax of noise robustness should disappear. The tax should persist for brightness-blind Gaussian augmentation tuned to the same robustness level. In two sentences: we think much of the robustness-accuracy trade-off for random noise comes from training on physically impossible corruptions. We test this by replacing them with exposure-consistent sensor noise and measuring the clean-versus-noisy accuracy frontier.

Core Claim

The claim concerns robustness to random, physically arising noise, not worst-case adversarial perturbations, and it is stated as a frontier comparison. Each training scheme traces a curve of clean accuracy against accuracy under real-like sensor noise as its strength parameter is varied. We predict that the exposure-consistent scheme's curve dominates the curve of uniform Gaussian augmentation. At equal accuracy under noise, it should lose essentially no clean accuracy.

The mechanism follows the error decomposition of Theoretically Principled Trade-off between Robustness and Accuracy. That paper splits robust error into natural error and "boundary error," which is the chance that an allowed perturbation pushes a point across the decision boundary. It then trades the two with a weight $\lambda$. Our reading is that boundary error for random noise is inflated when the perturbation set includes off-manifold images. Defending against those images consumes capacity without helping on real data. Exposure reduction is a perturbation set where label preservation holds for a physical reason: thinning the photons of a scene leaves the scene unchanged.

Research Program

The pipeline has three pieces, each reused or adapted from existing work. The first is the inverse camera pipeline from Unprocessing Images for Learned Raw Denoising. That paper inverts tone mapping, gamma, color correction, white balance and mosaicing, so that synthetic shot-and-read noise can be added in linear sensor space. It showed that denoisers trained this way beat those trained on sRGB noise by 14%–38% on real camera data. Brooks et al. stopped at restoration quality. We reuse their unprocessing to generate classifier training views and ask a question they did not ask, namely what this does to downstream clean accuracy.

The second piece is the exposure-scaled noise sample itself. Given a linear clean image $x$ and an exposure fraction $t \in (0,1]$, we draw

$$y_t = \frac{\alpha\,\mathrm{Poisson}(t x/\alpha) + n}{t}, \qquad n \sim \mathcal{N}(0, \sigma_g^2),$$

which has mean $x$ and per-pixel variance $\alpha x / t + \sigma_g^2 / t^2$. This is what a sensor with gain $\alpha$ and read noise $\sigma_g$ records after a $1/t$ brightness gain. We then re-apply the forward camera pipeline before feeding the result to the classifier. Here $t$ and $(\alpha, \sigma_g)$ are sampled per image from ranges matched to real sensors, as in the Brooks noise sampler.

The third piece is the training objective. Following TRADES, we use

$$\min_f \; \mathbb{E}\big[\mathcal{L}(f(x), y) + \lambda\, \mathbb{E}_t\, \mathrm{KL}\big(f(x)\,\|\,f(\mathrm{ISP}(y_t))\big)\big],$$

where $\mathrm{ISP}$ is the forward rendering and $\lambda$ sweeps the trade-off. The expectation over $t$ replaces TRADES' inner maximization over an $\epsilon$-ball, so training costs one extra forward pass per sample rather than a multi-step attack. We also run plain augmentation, with no consistency term, to see whether the objective matters at all.

We will analyze where the clean-accuracy tax comes from by splitting test error into the two TRADES components under each noise family. We will also check whether Gaussian-trained models lose clean accuracy mainly on images with large dark regions. Those are the regions where uniform noise departs most from the physics, and finding the loss concentrated there would directly support the mismatch explanation.

Baselines

The naive reference is standard training with no noise. It fixes the top of the clean-accuracy axis and the bottom of the noisy-accuracy axis.

The main competitor is uniform Gaussian augmentation applied directly to sRGB images, run at the same TRADES-style consistency weight. This is the obvious default, and it is the arm whose tax we claim is a mismatch artifact. For fairness, we sweep its standard deviation over a grid at least as fine as our $t$ range. We match per-image noise energy, measured as mean squared perturbation, between arms at each frontier point. Each arm gets the same epochs, architecture and number of hyperparameter trials.

The strongest existing method for the robustness-accuracy trade-off in the file is TRADES itself. It reports a smooth transition between 88.64% natural accuracy and 56.61% robust accuracy under PGD on CIFAR-10. We will retrain it under our pipeline and report its clean accuracy and its accuracy under physical noise. Pixel-reweighted adversarial training, from Improving Accuracy-robustness Trade-off via Pixel Reweighted Adversarial Training, shrinks the attack budget on background pixels chosen by class-activation maps. It reports a 1.2%–3.18% clean-accuracy gain on CIFAR-10 at about twice the training time of standard adversarial training. We include it as the strongest "non-uniform budget" competitor, run with its published schedule.

For noise augmentation evaluated on CIFAR-10-C noise corruptions, we found no reported result on Lacuna, so these numbers will be produced by us.

Two ablations isolate the key ingredient. The first applies Poisson-Gaussian noise directly in sRGB with the same $\alpha, \sigma_g$ ranges, skipping unprocessing. This separates the effect of modeling brightness dependence from the effect of applying it in the right domain. The second keeps the raw-domain pipeline but replaces brightness-dependent variance with a constant variance matched in mean. This isolates heteroscedasticity, meaning brightness-dependent noise variance, from the camera-pipeline round trip.

Beating the main competitor means one of two outcomes. Either our arm has higher clean accuracy at equal noisy accuracy, by a margin larger than the seed-to-seed spread over five seeds, at two or more frontier points. Or its frontier stays within that spread of the no-noise model's clean accuracy while matching Gaussian augmentation's best noisy accuracy.

Why not just…?

Why not just tune Gaussian augmentation strength, or TRADES' $\lambda$, more carefully? This is the real threat. A well-chosen small Gaussian standard deviation may already incur almost no tax. TRADES shows that $\lambda$ moves a model along one curve, and PART's gains suggest that the shape of the perturbation set can move the curve itself. Neither result establishes this for random sensor noise. The frontier comparison with matched tuning budgets settles the question directly. If the two frontiers coincide within seed variance, the mismatch hypothesis is wrong and we will report that.

Why not just apply a variance-stabilizing transform (Anscombe) and add Gaussian noise there? Done in linear raw space, adding Gaussian noise after an Anscombe transform and inverting approximately reproduces Poisson-Gaussian noise. We regard that as an implementation of the same idea, not an alternative to it. The approximation fails near zero photons, which is the dark-region regime that matters here. The file reports that the stabilize-then-denoise pipeline trails direct methods on real microscopy noise (32.71 vs 35.40 dB PSNR in A Poisson-Gaussian Denoising Dataset With Real Fluorescence Microscopy Images.). We will include a stabilized-Gaussian raw-domain arm and expect it to differ from ours mainly at the lowest exposures. A stabilizing transform applied to finished sRGB images ignores the camera pipeline, and our sRGB-Poisson ablation tests that case.

Why not just give adversarial training a brightness-dependent $\epsilon$ budget instead? We considered making this the main idea and rejected it. A per-pixel $\ell_\infty$ budget proportional to $\sqrt{\alpha x + \sigma_g^2}$ is, to first order, a uniform budget after a variance-stabilizing transform. It is therefore close to a reparameterization. It also quietly weakens the threat model, since an attacker is not bound by photon statistics. Any clean-accuracy gain would need a total-budget-matched control to rule out "smaller set, easier task." For these reasons we report $\ell_\infty$ PGD robustness only as a secondary measurement and make no adversarial claim.

Why not just use semantic masks as PART does? Class-activation masks depend on the model's own gradients. PART needs a 20-epoch burn-in and periodic mask updates, and it doubles training time. Exposure sampling needs no gradients. More importantly, semantic masks address adversarial budgets rather than physical noise. PART is still the best evidence that non-uniform perturbation sets help, so it stays as a baseline.

Proposed Experiments

On CIFAR-10 and ImageNet, we unprocess images with the Brooks pipeline, train each arm across its strength sweep, and evaluate on four inputs. These are clean test images, held-out exposure-scaled raw-domain noise, uniform Gaussian noise, and the noise corruptions of CIFAR-10-C. Testing on Gaussian noise checks that any frontier win is not just matched-noise overfitting. Our arm may plausibly lose there, and that loss would be an honest cost to report. We also report PGD and AutoAttack accuracy so readers can see whether physical noise training trades away any adversarial robustness relative to TRADES.

The medical setting is where the physics is most explicit. We use NIH ChestX-ray14 with the quantum and electronic noise ladder of Evaluating the Impact of Radiographic Noise on Chest X-ray Semantic Segmentation and Disease Classification Using a Scalable Noise Injection Framework. That paper found that classification was mostly resilient but that distinguishing Pneumothorax from Atelectasis lost 0.355 AUROC under quantum noise. We train classifiers with photon-count-reduction augmentation versus Gaussian augmentation. We then measure clean AUROC alongside AUROC across the ladder, and test whether the drop of 0.355 shrinks without lowering clean AUROC. That study also observed that intermediate noise slightly helps segmentation, which motivates a check for a regularization regime: at small exposure reductions, does our arm's clean accuracy exceed the no-noise model's?

Three results would be decisive. First, frontier dominance over Gaussian augmentation on ImageNet and ChestX-ray14 at matched tuning budgets. Second, the clean-accuracy tax of Gaussian-trained models concentrating on dark-region-heavy images, while ours shows no such concentration. Third, the sRGB-Poisson ablation landing between the two, which would show that the camera-pipeline round trip contributes independently. A sensitivity test with deliberately misspecified gain $\alpha$, off by factors of 2 and 4, will show how much exact calibration matters in practice. If the result is robust to misspecification, the method needs no per-sensor measurement. If it is not, the method requires sensor characterization.