Making a segmentation network robust to adversarial noise (tiny, deliberately chosen pixel changes that flip its predictions) is expensive. The standard recipe, adversarial training, repeatedly attacks every training image with several gradient steps through the whole network and then trains on the attacked images. For high-resolution dense prediction, where every pixel gets a label, this multiplies an already heavy training cost several times over. The best current shortcut starts from an image classifier that was already adversarially trained on ImageNet and fine-tunes it for segmentation. That shortcut still backpropagates the attack through the full high-resolution backbone at every step.
We propose a cheaper division of labor. Keep the robust backbone frozen, and adversarially train only the multi-scale context module: the block, such as atrous spatial pyramid pooling, that mixes information across many spatial ranges before the final per-pixel decision. The attack used during training is moved from the pixels to the backbone's output features, so each inner attack step backpropagates through the small head only. The feature-space attack budget is calibrated, per pyramid level, to how far real pixel attacks actually move the frozen robust features. The claim is that this recovers most of the robustness of full robust fine-tuning at a fraction of its training cost. We test that claim with the strongest available pixel-space evaluation. A secondary claim is that the multi-scale structure of the head is what makes this work: dilated branches with large receptive fields carry localized feature damage across the image, so this is where hardening pays off.
Background and the gap
Multi-scale context aggregation exists because dense prediction needs both large context and fine resolution. Dilated convolutions, which space out kernel taps to widen the receptive field without downsampling, give both. Dolz et al. replace pooling with dilation in IVD-Net to keep MRI boundaries sharp, and Baig et al. use dilation to combine local detail with global structure in Learning to Inpaint for Image Compression. Atrous spatial pyramid pooling runs several dilation rates in parallel. The Receptive Field Block assigns larger dilations to larger kernels, and Augmented ASPP uses non-monotonic dilation sequences to avoid "gridding," the sparse sampling gaps left by large dilations. All of these modules were designed for clean accuracy and speed. None was studied as the locus of adversarial vulnerability or as the place to spend adversarial-training compute.
Several papers motivate and constrain the proposal. Xie et al.'s dense adversary generation attacks segmentation and detection jointly over all pixels. Under its attack, FCN-VGG drops from 65.49% to 4.09% mIoU on Pascal VOC. Shuffling the perturbation spatially destroys the attack, which shows that dense vulnerability depends on spatially coherent structure, exactly the kind of structure context modules aggregate. That paper builds no defense. Croce, Singh and Hein supply both our evaluation and our main competitor. Their attack suite, SEA, combines a Jensen–Shannon loss, masked cross-entropy and a shrinking radius, and it exposes robustness overestimates of up to 17.2% left by earlier attacks. Their PIR-AT, robust fine-tuning from an adversarially trained ImageNet classifier, reaches up to 55.5% robust accuracy on ADE20K at $\epsilon = 4/255$ while training up to six times faster than adversarial training from scratch. PIR-AT updates the whole network and attacks through the whole network. We keep its key insight, that robust backbones transfer, and remove its remaining cost.
Parameter-efficient defenses also constrain the design. Robustness Tokens freezes a non-robust ViT and trains ten prefix tokens. That recovers DINOv2-G classification from 0.3% to 63.1% under attack, but ADE20K mIoU under attack rises only from 4.7 to 23.3. Dual [CLS] adapters reach 49.87% robust ImageNet accuracy with 768 extra parameters. That method is classification-only and has no handle on dense spatial features. We read the ADE20K result of Robustness Tokens as a negative result: freezing a non-robust backbone and adding a few global parameters does not protect dense outputs. For that reason we never freeze a non-robust backbone. The frozen backbone must itself be robust, and the trained part must be spatial.
Core Claim
Let $z_\ell(x)$ be the frozen robust backbone's feature map at pyramid level $\ell$ for image $x$. Let $h_\psi$ be the multi-scale context module plus decoder with parameters $\psi$, and let $\mathcal{L}$ be the per-pixel segmentation loss against label map $y$. We train
$$\min_\psi \; \mathbb{E}_{(x,y)} \max_{\|\eta_\ell\|_{2,\text{loc}} \le \rho_\ell \;\forall \ell} \mathcal{L}\big(h_\psi(\{z_\ell(x) + \eta_\ell\}), y\big),$$
where $\eta_\ell$ is a feature perturbation and $\|\cdot\|_{2,\text{loc}}$ bounds the channel-wise $\ell_2$ norm at each spatial location. Bounding per location, rather than globally, lets the inner attacker concentrate damage on small regions, as real dense attacks do. The radius $\rho_\ell$ is a high quantile of the per-location displacement $\|z_\ell(x+\delta) - z_\ell(x)\|$ measured under real pixel-space SEA perturbations $\delta$ on a small calibration set. We re-measure it a few times during training. The inner maximization is a few projected gradient steps that backpropagate through $h_\psi$ only. The backbone runs one forward pass per image, with no stored activations for parameter gradients and no optimizer state.
What would be new is the combination of a robust frozen backbone, a feature-space inner loop calibrated to real pixel attacks, and training restricted to the multi-scale aggregator. We also test, rather than assume, whether the dilated multi-branch structure is what makes head-only hardening sufficient.
Research Program
The base models are DeepLab-style networks with an atrous spatial pyramid pooling head, on two backbones: an adversarially trained ImageNet ConvNet, and an adversarially trained ViT with a pyramid adapter. We use the robust ImageNet checkpoints in the way PIR-AT uses them. As a secondary architecture we use an RFB-style detection head on Pascal VOC, to check that the effect is not tied to segmentation.
The work has three parts. The first is a diagnostic. On a PIR-AT-style model, we measure where pixel attacks do their damage by comparing feature displacement at each backbone level and inside each dilation branch. We also check whether repairing the head features alone, by swapping in clean head inputs at one level at a time, restores predictions. This tells us whether the head is a plausible place to intervene before we build anything. The second part is the training method above, with honest compute accounting: wall-clock time, GPU-hours, and peak memory on identical hardware. The head can be expensive at output stride 8, so savings are measured, not assumed. The third part asks whether multi-scale structure matters. We compare a single shared radius with per-branch radii, and we compare monotonic dilation sequences with the non-monotonic, gridding-free sequences of Augmented ASPP. The second comparison tests whether sparse sampling gaps make heads harder to harden.
The main risk is that feature-ball robustness does not imply pixel robustness. Real attack displacements are highly structured, and a ball is a loose surrogate, which could leave holes or impose an excessive clean-accuracy cost. We address it directly. A hybrid variant runs one pixel-space attack step through the frozen backbone every $N$ iterations, and the dose-response curve between pure feature-space and full pixel-space training is itself a result.
Baselines
The do-nothing reference is the frozen robust backbone with a head trained only on clean images. This is PIR-AT's "standard fine-tuning" option restricted to the head. It shows how much robustness the backbone alone gives, and any gain we report must be measured above it.
The strongest existing method is full PIR-AT, whose reported strongest number is up to 55.5% robust accuracy on ADE20K at $\epsilon = 4/255$. We rerun it ourselves from the same robust checkpoint, on the same data and schedule length, with the same hyperparameter search budget. We report robust mIoU and robust pixel accuracy under SEA alongside clean mIoU, because 55.5% is a pixel-accuracy figure. Beating it means matching its SEA robust mIoU within one point while using at most a third of its GPU-hours. Exceeding it at any cost would also count, though we do not expect that.
The parameter-efficient reference is Robustness Tokens, which reports 23.3 mIoU under white-box attack on ADE20K. Because that number came from a different attack and backbone, we rerun it on our ViT backbone under SEA.
The single-step reference is the Fast Newton Method of Single-Step Adversarial Training for Semantic Segmentation, run on full pixel space from the robust checkpoint. Lacuna lists no reported result for FNM on Cityscapes or Pascal VOC in absolute terms, so its numbers will come from our reruns.
The ablation that isolates the key ingredient is head-only adversarial training with pixel-space attacks through the frozen backbone. It has the same trainable parameters as our method and the full inner-loop cost, so it separates "which parameters are trained" from "where the attack lives." A second ablation trains an equal number of parameters in the last backbone stage instead of the context module, testing whether the multi-scale head is special or just a convenient parameter count.
Why not just…?
Why not just run PIR-AT end to end? It is the real threat, and it may simply win on robustness. Its cost, though, is a full-network attack and full-network update at every step, and our method targets exactly that cost. The head-to-head under matched data and SEA settles the question. If the frozen-backbone variant loses more than a few points, the honest conclusion is that robustness has to be distributed through the backbone during downstream adaptation, and we would report that.
Why not just freeze the backbone and train robustness tokens or adapters? The Robustness Tokens paper's own ADE20K result, 23.3 mIoU under attack, shows that global tokens on a non-robust frozen backbone leave dense outputs exposed. Dual [CLS] adapters have no spatial parameters at all. Our rerun of Robustness Tokens under SEA on a common backbone makes the comparison direct.
Why not just use input randomization at test time? Random resizing and padding works against fixed attacks on classifiers, but its authors note that it is bypassed by attacks that average gradients over the random transforms. It also misaligns predicted and ground-truth pixel grids. We include it in the evaluation with an expectation-over-transformation version of SEA so that the comparison does not credit gradient masking.
Why not just do cheap single-step pixel-space training with FNM? FNM still backpropagates through the whole high-resolution network. Its single step size is averaged over all pixels, which Croce et al.'s analysis of pixel-dominated losses suggests is fragile, and it was never evaluated under SEA. Running FNM from the same robust checkpoint, with matched GPU-hours, shows whether a single full-network step beats several head-only steps.
Proposed Experiments
The primary benchmark is ADE20K, with Pascal VOC 2012 and Cityscapes as secondary segmentation benchmarks and Pascal VOC 2007 detection with an RFB head as a cross-task check. Robustness is measured under SEA at $\epsilon \in \{2/255, 4/255, 8/255\}$ with $\ell_\infty$ attacks run end to end through the whole frozen-plus-trained model. Our training-time attack is in feature space, but every evaluation is in pixel space, so feature-space robustness cannot be mistaken for real robustness. We report clean mIoU, robust mIoU and robust pixel accuracy, and per-class robust IoU split by object size, since small objects are where averaged or global defenses are expected to fail. Training GPU-hours and peak memory are reported for every method.
Three decisive checks follow. The first is whether calibrated feature-space head training lands within one robust-mIoU point of our PIR-AT rerun at a third or less of its compute. If it lands far below, while pixel-space head-only training does not, the feature-ball surrogate is the failure point and the hybrid dose-response curve shows how much pixel-space attacking is needed. The second is whether training the context module beats training an equal-sized slice of the backbone. If it does not, the "multi-scale head is the right place" claim fails even if the method is cheap. The third is whether per-branch radii and gridding-free dilation sequences measurably help small-object robust IoU, which would tie the result to multi-scale aggregation rather than to generic head capacity.
References
- IVD-Net: Intervertebral Disc Localization and Segmentation in MRI with a Multi-modal UNet
- Learning to Inpaint for Image Compression
- Receptive Field Block Net for Accurate and Fast Object Detection
- Fully Convolutional Neural Network with Augmented Atrous Spatial Pyramid Pool and Fully Connected Fusion Path for High Resolution Remote Sensing Image Segmentation
- Adversarial Examples for Semantic Segmentation and Object Detection
- Robust Semantic Segmentation: Strong Adversarial Attacks and Fast Training of Robust Models
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- Revisiting adapters with adversarial training
- Single-step Adversarial Training for Semantic Segmentation
- Mitigating Adversarial Effects Through Randomization