Segmentation models trained on driving and scene-parsing data lean heavily on where things usually are. Road sits at the bottom of the frame and sky at the top. Attackers exploit exactly this. Perturbing the sky can erase a car from a depth map, and attacked segmenters collapse into painting the whole frame with one background class. We propose to turn that weakness into a source of efficiency.
Adversarial training is the main defense: the model is trained on worst-case perturbed inputs. Its cost is an inner search that must find a perturbation for every image at every step. Our hypothesis is that, because the models rely on shared layouts, the perturbations that fool them are also largely shared across images. They line up with image position and with the classes present in the scene. If so, a small running "perturbation prior", indexed by position and by ground-truth class, can serve as a strong starting point for a single cheap attack step. That would give the robustness of multi-step training at roughly single-step cost, without the large per-image memory buffers that current fast methods need.
Core Claim
Fast adversarial training replaces the multi-step search with one gradient step. On its own, one step tends toward catastrophic overfitting: the model becomes robust to the one-step attack while multi-step attacks still break it almost completely. The best known fix is to start the one-step attack from a remembered perturbation. Prior-Guided Adversarial Initialization for Fast Adversarial Training (PGI) does this by recycling the perturbation from the previous batch, the previous epoch, or a momentum average over epochs. It reports 55.18% PGD-10 robust accuracy on CIFAR-10 versus 53.76% for full multi-step training.
The "prior" in PGI is a per-image record of past perturbations. For dense prediction that record is expensive. A full-resolution Cityscapes image is 3×1024×2048 values, about 25 MB in 32-bit floats, so a buffer for the 2,975 training images is roughly 75 GB. Random cropping also misaligns stored perturbations with the image regions they were computed on.
Our claim is that for segmentation the useful part of this memory is not image-specific. It is a layout-conditioned component that can be stored once for the whole dataset: a low-resolution map indexed by image position plus a small perturbation tile per class. This component can be placed onto any training crop using the ground-truth label map, which is available for free during training. The new element is that the initialization is shared across images and conditioned on layout. Prior fast methods are either per-image (PGI) or uninitialized (the Fast Newton Method below).
Why the Spatial-Prior Literature Predicts This Should Work
Two papers motivate the hypothesis directly. Targeted Adversarial Perturbations for Monocular Depth Prediction shows two things about depth networks trained on KITTI. First, they encode a strong vertical prior. Second, effective attacks place perturbation mass in contextually linked regions far from the target object. Both suggest that attack structure follows scene layout rather than individual image content. That paper only studies attacks, at about 12 seconds per frame, and never trains a defense.
Robust Semantic Segmentation: Strong Adversarial Attacks and Fast Training of Robust Models observes that attacked standard segmenters collapse into whole-frame layout predictions such as uniform "sky" or "grass". This is again a sign that attacks push the model into a shared, prior-driven failure mode.
Explicit-prior models show why this pathway is strong. Convolutional Patch Networks with Spatial Prior for Road Detection and Urban Scene Understanding raises KITTI road MaxF from about 76% to 86.50% just by feeding in pixel coordinates. IDDA documents layout-driven misclassification under urban-to-rural shift, with mIoU as low as 14.27% without adaptation. These papers motivate the hypothesis but constrain nothing about the method itself.
Research Program
The first stage is a diagnostic that can kill the idea cheaply. We take a standard, non-robust segmenter on Cityscapes and compute strong per-image attacks on held-out images. We then ask how much of each attack's loss increase is recovered by three shared perturbations, each estimated from training images:
- the dataset-mean perturbation in image coordinates (a universal-perturbation-style baseline);
- the class-conditioned composite (position map plus per-class tiles laid out by the label map);
- a random sign perturbation of equal norm.
The attacks are run with the Semantic Ensemble Attack (SEA) from Croce et al. Robust Semantic Segmentation, so that cross-entropy attacks do not underestimate vulnerability. We repeat this measurement for partially robust models during training, because the shared component may shrink as the model becomes robust. If the class-conditioned composite does not recover substantially more of the loss increase than the position-only mean, the class-conditioning ingredient is unjustified and we drop it.
The second stage is the training method. We keep a position map $P \in \mathbb{R}^{3 \times h \times w}$ at low resolution and per-class tiles $T_c \in \mathbb{R}^{3 \times p \times p}$ for each class $c$. Both are updated as exponential moving averages of the final perturbations found during training. For a training crop $x$ with label map $y$, the attack starts from
$$\delta_0 = \Pi_\epsilon\Big(\mathrm{up}(P)[\text{crop}] + \sum_c \mathbb{1}[y = c] \odot \mathrm{tile}(T_c)\Big).$$
Here $\Pi_\epsilon$ clips to the $\ell_\infty$ budget $\epsilon$, $\mathrm{up}$ upsamples to full resolution before cropping, $\mathbb{1}[y=c]$ is the pixel mask of class $c$, and $\mathrm{tile}$ repeats the tile across the image. From $\delta_0$ we take one signed gradient step. We borrow from PGI its regularizer that ties the model's outputs on the initialized and final perturbed inputs.
The per-step cost equals one-step adversarial training plus a negligible gather operation. Memory is $O(hw + Cp^2)$ instead of $O(N \cdot HW)$, where $N$ is the number of training images and $H \times W$ the full image size. The step size can either be fixed or taken from the analytical rule of Single-Step Adversarial Training for Semantic Segmentation. That paper's Fast Newton Method (FNM) computes one scalar step per image from a Taylor approximation of a pixel-averaged gain. Testing both lets us see whether initialization and step-size control are complementary.
The third stage is a secondary analysis that bridges back to spatial prior bias. We do not claim that this method reduces layout rigidity. We will, however, measure clean and attacked mIoU on vertically flipped Cityscapes images and on IDDA cross-town splits. This tells us whether robustness bought through a layout-shared initialization makes models more tied to canonical layouts or less. Either result is informative. A model that is robust only in canonical layouts would be a warning about all fast methods in this regime.
Baselines
The naive reference points are an undefended model and plain one-step training with a random start. The first sets the floor. The second shows whether catastrophic overfitting appears in our setup, which we check by tracking multi-step robust mIoU across epochs. On PASCAL VOC with PSPNet, Dynamic Divide-and-Conquer Adversarial Training reports 5.4% mIoU under BIM (the Basic Iterative Method, a multi-step attack) with no defense. We reproduce that setting so our numbers are comparable.
The quality ceiling is multi-step adversarial training. The same paper reports 27.9% BIM mIoU for standard multi-step training (clean 74.3%). Its divide-and-conquer variant (DDC-AT) reaches 33.8% (clean 76.0%). DDC-AT is the strongest reported robust result on this setting in the file, but it pays the full multi-step training cost. We do not need to beat 33.8% to succeed. Success means matching standard multi-step training's robust mIoU at no more than about 1.5 times the cost of one-step training, and getting within a few points of DDC-AT at a fraction of its wall-clock time.
The strongest direct competitor on the efficiency question is FNM. It is single-step, needs no memory, and is reported to match or exceed multi-step BIM training at small budgets on Cityscapes and VOC. No reported FNM mIoU figure was found on Lacuna, so we will reimplement it on identical backbones, crops, epochs, and budgets.
The second direct competitor is PGI in two variants. The previous-batch variant has tiny memory. The momentum variant requires per-image buffers, which we will run at reduced resolution where full-resolution buffers do not fit, and report what had to be compromised.
On ADE20K we compare against PIR-AT, which fine-tunes from an adversarially pre-trained ImageNet backbone. It reports robust accuracy up to 55.5% at $\epsilon = 4/255$ (Robust Semantic Segmentation). We also run PIR-AT with our initialization substituted for its multi-step inner loop, since the two are compatible.
All methods get the same data, the same augmentation, a matched number of training epochs, and the same small step-size and $\epsilon$ tuning grid. We report wall-clock time and peak memory alongside mIoU.
Ablations isolate the key ingredient. We compare the full composite against the position map alone, the class tiles alone, and a shuffled-label control in which tiles are placed using another image's label map. The shuffled control tests whether layout alignment, rather than simply having some nonzero initialization, is what matters.
Why not just…?
Why not just use FNM, which already makes single-step segmentation training stable? FNM controls how large the one step is, not where it starts. Its scalar step averages the gain over all pixels. That stabilizes training but discards the spatial structure the diagnostic is designed to measure. It is a genuine threat. If FNM alone matches our robust mIoU at equal cost, the initialization adds nothing. The FNM-only versus FNM-plus-initialization comparison settles this directly.
Why not just use PGI's previous-batch initialization, which needs no large buffer? This is the most serious alternative. In heavily layouted driving data, the previous image's perturbation may already carry most of the shared structure, making it a crude, implicit version of our position map. Our method differs in two ways: it averages over many images rather than inheriting one, and it is aligned to the current image's class layout. The ablation ladder of previous-batch, position-only mean, and full composite determines whether either difference matters. We will report a negative result if the previous-batch variant is equivalent.
Why not just fine-tune from a robust backbone as PIR-AT does? PIR-AT reduces the number of epochs needed but still runs adversarial training on the target dataset, so its per-step cost is unchanged. Our method reduces the per-step cost, so the two compose. The relevant experiment is PIR-AT with and without our initialization at matched wall-clock time.
Why not just perturb positional embeddings, which is nearly free? AdPE perturbs position coordinates with about 1.5% overhead. However, it was evaluated only for clean downstream accuracy. Perturbing positions also places no bound on pixel-space attacks, so there is no reason to expect it to deliver $\ell_\infty$ robustness. We include an AdPE-style fine-tune as a cheap control, expecting it to fail under SEA evaluation.
Proposed Experiments
The decisive experiments run on PASCAL VOC with PSPNet (comparable to the DDC-AT numbers), Cityscapes (comparable to FNM), and ADE20K (comparable to PIR-AT). We use $\ell_\infty$ budgets of 4/255 and 8/255. Robustness is measured with both BIM and SEA, and any method whose gain over the baselines appears only under BIM is treated as unreliable.
The primary metrics are clean mIoU, robust mIoU, training wall-clock time, and peak GPU memory. A secondary metric is mIoU under vertical flip and IDDA cross-town shift.
The idea is supported if three things hold. First, the diagnostic shows that the class-conditioned shared perturbation recovers clearly more attack loss than the position-only mean. Second, our method avoids catastrophic overfitting over full training. Third, it beats both FNM and previous-batch PGI in robust mIoU at matched cost.
The idea is refuted if previous-batch PGI or FNM matches it. In that case the contribution narrows to the diagnostic finding about how attack structure is shared across images, which we would still report.
References
- Prior-Guided Adversarial Initialization for Fast Adversarial Training
- Targeted Adversarial Perturbations for Monocular Depth Prediction
- Robust Semantic Segmentation: Strong Adversarial Attacks and Fast Training of Robust Models
- Convolutional Patch Networks with Spatial Prior for Road Detection and Urban Scene Understanding
- IDDA: a large-scale multi-domain dataset for autonomous driving
- Single-step Adversarial Training for Semantic Segmentation
- Dynamic Divide-and-Conquer Adversarial Training for Robust Semantic Segmentation
- AdPE: Adversarial Positional Embeddings for Pretraining Vision Transformers via MAE+