Almost every deployed vision model receives a large image and shrinks it before classification. A 672-pixel photo, for example, is shrunk three-fold to 224 pixels. Attackers can exploit this shrinking step. They hide a different image, or a backdoor trigger, in exactly the pixels the resizer will keep. They can also spread adversarial noise across the large image so that it concentrates after resizing. This note makes a simple argument. The resizer is a fixed linear map, so the damage an attacker can do to the large image is exactly the damage they can do to the small image within a rescaled budget. The rescaling factor is set by one number: how much the resizer can amplify a perturbation. We call this the resizer's amplification factor. Common default resizers amplify perturbation energy by up to the downscaling factor. Averaging resizers amplify by nothing.
This matters because the amplification is currently paid for in the most expensive way. It is either ignored, leaving pipelines broken, or absorbed by adversarial training at a larger radius. Adversarial training here means training on worst-case perturbed inputs, which costs roughly ten times more than ordinary training and lowers clean accuracy. We propose to measure this hidden cost across real resizers and datasets. We will then show that picking a non-amplifying resizer lets cheap low-resolution defenses carry over to high-resolution attacks, with a guarantee and without extra training. Such defenses include adversarial training at 224 pixels, or a few learned tokens on a frozen backbone. The result also gives them a smaller training radius, and hence better clean accuracy, for the same protection.
Core Claim
Let $S$ be the linear downscaling operator mapping a high-resolution image with $k^2 N$ pixels to a low-resolution image with $N$ pixels, where $k$ is the scale factor. A high-resolution perturbation $\delta$ reaches the model as $S\delta$. Measure budgets as per-pixel root-mean-square size $\sigma$, since that is what governs visibility at either resolution. Then a high-resolution perturbation of per-pixel size $\sigma$ arrives at the model with per-pixel size at most $a(S)\,\sigma$, where
$$a(S) = k\,\|S\|_2 ,$$
and $\|S\|_2$ is the largest factor by which $S$ can stretch an image's $\ell_2$ norm. For area averaging, $SS^\top = I/k^2$. So $\|S\|_2 = 1/k$ and $a = 1$, and the image of the high-resolution ball is exactly the low-resolution ball of the same per-pixel radius. For nearest-neighbour sampling, $\|S\|_2 = 1$ and $a = k$: the attacker puts all their energy on the surviving pixels. Bilinear or bicubic resizing without anti-aliasing touches only a small fixed neighbourhood per output pixel, which places it between these extremes, with $a$ on the order of $k/2$.
The same algebra explains a finding reported without explanation. Rethinking Image-Scaling Attacks found that $\ell_\infty$ attacks gain little from scaling. Any resizer with nonnegative weights summing to one has $\ell_\infty$ gain exactly one. Bicubic and Lanczos kernels, which have negative lobes, have $\ell_\infty$ gain slightly above one.
The claim with teeth concerns the robustness-accuracy trade-off. At a fixed high-resolution threat $\sigma$, a pipeline with amplification $a$ needs a low-resolution model robust at radius $a\sigma$. A defender using a default amplifying resizer must therefore train at up to $k$ times the radius, and pay the corresponding clean-accuracy and compute cost. Switching to a resizer with $a = 1$ recovers that cost at training price zero. Camouflage and scaling-based poisoning fall under the same bound. To move the downscaled image by per-pixel amount $\Delta$ through an $a = 1$ resizer, the attacker must change the large image by at least $\Delta$ per pixel, so the change is no longer hidden.
Relation to Prior Work
Three papers motivate the threat. Seeing is Not Believing introduced camouflage images that downscale into a different target. Backdooring and Poisoning Neural Networks with Image-Scaling Attacks turned camouflage into clean-label poisoning, reaching near-100% attack success with 5% poisoned data. Rethinking showed that joint scaling-plus-adversarial attacks are much stronger than attacks on the small image alone. Its Scaling-aware Noise Sampling (SNS) samples noise at low resolution and projects it to high resolution. On CelebA at 10K queries it reaches 60.0% success versus 10.8% for a standard attack, and it bypasses 4 of 5 scaling defenses. That paper studies attacks only. We supply the defender-side accounting: amplification is a property of $S$ that can be computed in advance, and its size determines how much robust training a pipeline actually needs.
Two lines of work constrain the design. On the Detection of Image-Scaling Attacks detects global scaling attacks perfectly via Fourier peaks, but localized triggers only about 90% of the time. Its rejections also deny service to odd benign images, so we treat detection as a complement rather than a fix. Mitigating Adversarial Effects Through Randomization randomizes resizing over a narrow range and admits that the defense is open to Expectation Over Transformation (EOT) attacks. EOT means attacking the average over the random transformations. Rethinking found randomized filtering to be the only scaling defense that held up. Our analysis predicts why: it is non-amplifying in expectation. It also predicts that this holds only in expectation, which is testable.
Two papers supply the downstream defenses we plug in. One is ordinary low-resolution adversarial training. The other is Robustness Tokens, which trains about 10 extra input tokens on a frozen transformer in 200 steps. It restores DINOv2-G white-box accuracy from 0.3% to 63.1%, with clean accuracy of 85.2% against 85.6% for the plain backbone. That work only ever attacked 224-pixel inputs. Our reduction states exactly when such a defense carries to high-resolution inputs and at what radius.
Research Program
The work has three parts.
Audit of resizers. We compute $a(S)$ numerically, by power iteration on $S^\top S$, for the resizers practitioners actually call. These include nearest, bilinear, bicubic, area and Lanczos, each with and without anti-aliasing. Because library defaults differ in whether they anti-alias, the audit records the default behaviour of the common imaging and tensor libraries. We cover scale factors from 2 to 10, and handle the randomized-filtering resizer by computing its amplification per random draw and in expectation.
Empirical reduction. We attack the full pipeline (high-resolution image → $S$ → model) with three attacks: white-box projected gradient descent through $S$, an EOT variant for randomized resizers, and SNS-style black-box queries. We compare the resulting robust accuracy to the low-resolution robust accuracy at radius $a(S)\sigma$. Strict linearity is broken in two places. Rounding to 8-bit after resizing is one; we measure its effect. Clipping to the valid pixel range only shrinks the attacker's set, so it does not break the bound.
Pareto study. For each resizer, we train low-resolution defenses across a range of radii. We then plot clean accuracy against high-resolution robust accuracy and training compute. The point is to show how far the frontier moves when only the resizer changes.
Baselines
The naive reference is a standard ResNet-50 on ImageNet and a ResNet-34 on CelebA, matching the evaluation setup of Rethinking, together with a frozen DINOv2 backbone. All use the library-default bilinear resizer with no defense. This is the pipeline Rethinking attacks, and it fixes the amplification we expect defenders to be silently paying.
The strongest scaling-specific competitor is randomized filtering, the only scaling defense Rethinking found robust. It is strongest because it is the only resizer already shown to survive SNS. We run it with the same backbones and the same attack budgets, plus EOT with matched query counts. Beating it means equal or better high-resolution robust accuracy under EOT with higher clean accuracy. We expect to win on clean accuracy and on variance, not necessarily on robustness.
The strongest model-side competitor is Robustness Tokens on frozen DINOv2. Its reported 224-pixel numbers are the anchor: 63.1% white-box robust accuracy and 85.2% clean. We retrain the tokens under the same 200-step budget once per resizer, and evaluate under high-resolution attacks at matched per-pixel budget. Success means the area-resizer pipeline matches the tokens' low-resolution numbers at high resolution, while the default-bilinear pipeline falls well short at the same $\sigma$. We also include Xie et al.'s random resize-and-pad. It reports DeepFool accuracy recovered from 1.1% to 98.2% with clean accuracy going from 100% to 97.3% on their clean subset, and a NIPS 2017 challenge score of 0.924 combined with adversarial training versus 0.773 for adversarial training alone. We run it on the same images under EOT.
The expensive upper reference is adversarial training directly at high resolution through $S$, at matched epochs. For high-resolution adversarial training of these backbones, no reported result was found on Lacuna. It costs about $k^2$ more per step. Our reduction predicts that with an area resizer it buys nothing over low-resolution training at the rescaled radius.
The key ablation isolates amplification. We keep the backbone, weights and defense fixed and change only the resizer: bilinear without anti-aliasing versus with it, and nearest versus area. Any change in high-resolution robustness should then track the change in $a(S)$ alone.
Why not just…?
Why not just switch to area resizing and stop? For camouflage and scaling-based poisoning, this is largely our recommendation, and we say so. The research question is what remains. A non-amplifying resizer leaves the model exactly as fragile as it is at low resolution: a standard model is still broken at radius $\sigma$. So it must be combined with a low-resolution defense. A model trained with one resizer and deployed with another may also lose clean accuracy. We measure that drop, and how much brief retraining or token tuning recovers it. If the drop is negligible and robustness transfers as predicted, the practical message is short. The trade-off accounting, which shows how much radius and clean accuracy amplifying pipelines waste, is the contribution.
Why not just use randomized filtering, since it already survived? Rethinking shows it survives SNS. Our analysis says this is because it is non-amplifying on average, but a single draw can behave like nearest-neighbour sampling. Two costs follow: prediction variance that lowers clean accuracy, and per-draw worst cases that an attacker targeting single queries might exploit. Comparing it with the area resizer under EOT, at equal clean accuracy, settles whether its randomness buys anything that deterministic averaging does not.
Why not just blur, JPEG-compress, or diffusion-purify the input? APBench shows that lossy operations strip fragile noise, and that diffusion purification restores poisoned-data accuracy to 80–92%. But blur and JPEG are nonlinear and can be broken by adaptive attacks: Rethinking bypasses median filtering, and MetaCloak survives blurring by attacking the average over transformations. Diffusion adds many denoising passes per image at inference time. We include JPEG and median filtering with adaptive attacks as baselines. We drop diffusion except for one cost-matched comparison on a 1,000-image subset.
Why not just adversarially train at high resolution? This is the honest brute-force answer, and we run it as the upper reference. If it clearly beats low-resolution training at radius $a(S)\sigma$ under an area resizer, our reduction is missing something in practice, most likely rounding effects or attack-optimization gaps. That outcome would be a reportable negative result.
Proposed Experiments and Decisive Checks
The datasets are 1,000 ImageNet images resized to 672 pixels and fed to models at 224 ($k = 3$), following Rethinking, and 1,000 CelebA images for the black-box SNS setting. To test large scale factors, where amplification should matter most, we add the SelvaBox high-resolution aerial tiles, downscaled by up to $k = 10$ for a tree-crown classification proxy. Metrics are clean accuracy, high-resolution robust accuracy at per-pixel budget $\sigma$ (in $\ell_2$ and $\ell_\infty$), attack success at fixed query budgets, and GPU-hours of training. We also re-run camouflage and adaptive histogram-matching poisoning. For those we measure attack success together with the per-pixel high-resolution change the attack needs.
Three checks decide the proposal. First, the predicted curve must hold: high-resolution robust accuracy should match low-resolution robust accuracy at $a(S)\sigma$ within attack noise for every resizer. In $\ell_\infty$, all nonnegative-weight resizers should coincide. Second, the trade-off must move: at matched high-resolution robust accuracy, the area pipeline should show higher clean accuracy and lower training compute than the default-bilinear pipeline, which needs a larger radius to cover the same threat. If that gap is small at $k = 3$ but large at $k = 10$, the effect is real but matters mainly for high-resolution sensors. Third, camouflage must become visible: under an area resizer, camouflage poisons should require per-pixel high-resolution changes equal to the low-resolution target change, so that the Fourier-peak detector and plain visual inspection both catch them.
References
- Rethinking Image-Scaling Attacks: The Interplay Between Vulnerabilities in Machine Learning Systems
- Seeing is Not Believing: Camouflage Attacks on Image Scaling Algorithms
- Backdooring and Poisoning Neural Networks with Image-Scaling Attacks
- On the Detection of Image-Scaling Attacks in Machine Learning
- Mitigating Adversarial Effects Through Randomization
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- APBench: A Unified Availability Poisoning Attack and Defenses Benchmark
- MetaCloak: Preventing Unauthorized Subject-Driven Text-to-Image Diffusion-Based Synthesis via Meta-Learning