Spending the Adversarial Training Budget Where Volumetric Segmentations Break

Adversarial training makes a network robust by repeatedly attacking it during training and learning from the attacks. It is expensive because every training step first runs an inner loop of several gradient-based attack iterations. For 3D medical segmentation networks, which label every voxel of a CT or MRI volume, this cost is hard to afford. It is also mostly wasted. A training patch from an abdominal CT is largely air, bone and organ interiors that an attack cannot flip, plus a thin band of ambiguous voxels at organ and lesion margins where predictions actually change. We propose to measure, for each training patch, how much of its foreground is close to flipping, and to give each patch a number of attack iterations in proportion to that measurement. The measurement is tracked across training as a moving average, so it reflects the patch's history rather than one noisy snapshot. The loss on the resulting adversarial patch is then reweighted voxel by voxel toward the voxels the attack actually moved.

The motivation is concrete. Frequency Domain Adversarial Training for Robust Volumetric Medical Segmentation shows that an attack perturbing 3D frequency coefficients (VAFA) cuts UNETR's Dice on Synapse multi-organ CT from 74.3% to 32.2%. Their defense, VAFT, runs this attack uniformly on every patch at every step and recovers 66.33% Dice under attack. The open question is whether most of that robustness can be kept at a fraction of the attack compute. Our answer rests on one specific claim: the right signal for allocating attack effort in dense prediction is the tail of the per-voxel margin distribution inside a patch, not a patch-averaged confidence. Averaging is exactly what breaks when image-level methods are ported naively to segmentation.

Core Claim

In classification, compute-aware adversarial training works by sorting whole images by how close they sit to the decision boundary. BulletTrain places images into three bins using a signed prediction variance. It gives full attack iterations only to "boundary" images and reports a 2.2x speedup for TRADES and MART on CIFAR-10 with no accuracy loss. Geometry-aware Instance-reweighted Adversarial Training (GAIRAT) instead upweights images that need few attack steps to flip, but it still attacks every image fully, so it saves no compute. Both treat the image as the unit of difficulty.

A segmentation patch is not one unit. If one averages voxel confidences over a patch, background dominates, and nearly every patch looks "robust." We claim that a patch's attack budget should be set by the fraction of its foreground voxels whose clean margin is small. Here "margin" means the gap between the logit of the correct class and that of the strongest competitor. We further claim that this fraction, smoothed over epochs, predicts which patches the frequency attack will damage well enough to cut inner-loop cost substantially at matched robust Dice.

We treat the relationship between spatial boundaries (anatomical contours) and decision boundaries (small-margin voxels) as an empirical question, not an assumption. If the distance to the ground-truth contour predicts attack damage as well as the model's own margin does, the cheaper label-derived signal should win. The program is built to find that out.

Research Program

The first component is a measurement study, run before any method is built. We train UNETR and UNETR++ on Synapse and ACDC with uniform VAFT, following the protocol of Frequency Domain Adversarial Training. At regular checkpoints we record four quantities for every training patch: the per-voxel clean margin, the distance of each voxel to the nearest ground-truth contour, the attack step at which each voxel flips (a voxel-level analogue of GAIRAT's step count), and the Dice drop the patch suffers under a full attack. From these we answer three questions. Are the voxels that flip concentrated at contours, or do low-contrast organ interiors also flip under frequency perturbations? How well does each candidate statistic rank patches by attack damage? How stable are those rankings from epoch to epoch? These answers set the design and are publishable on their own, since no work in the file maps spatial contour uncertainty onto adversarial vulnerability.

The second component is the training method. After the clean forward pass that training already performs, we compute a vulnerability score for each patch $p$. Let $s_p$ be an exponential moving average, across the epochs in which patch $p$ has been sampled, of the fraction of its foreground voxels whose clean margin is below a threshold $\tau$. Tracking this history is the method's use of the optimization trajectory, and the moving average follows BulletTrain's own tracking scheme. With an average budget of $B$ attack iterations per patch and a maximum of $K$, patch $p$ receives

$$k_p = \min\!\Big(K,\ \mathrm{round}\big(B \cdot s_p^{\gamma} / \overline{s^{\gamma}}\big)\Big),$$

where $\overline{s^{\gamma}}$ is the batch mean of $s_p^{\gamma}$ and $\gamma$ controls how sharply compute concentrates. This continuous rule is the allocation scheme BulletTrain explicitly left to future work. Because VAFA already operates on 3D patches through their frequency transform, patch-level allocation fits the attack naturally.

Real savings require that finished patches actually leave the computation. Patches drop out of the active batch once their $k_p$ iterations are done, so later forward and backward passes run on a physically smaller tensor rather than a masked one. We count cost in measured GPU-hours, not iterations. In the outer loss, each voxel's Dice and cross-entropy terms are weighted by $1 + \lambda\,\mathbb{1}[\text{margin fell below } \tau \text{ under the attack}]$, so the model learns most from voxels the attack moved. VAFT's frequency consistency loss is kept unchanged.

The third component checks for the known failure modes of reweighting. GAIRAT is reported to overfit to the attack it uses to compute weights, losing robustness under other attack geometries. We therefore evaluate against spatial PGD as well as VAFA. We also run an adaptive attack that concentrates its perturbation on the patches and regions our allocator starved, to test whether low-budget patches became an exploitable blind spot.

Baselines

The naive reference points are an undefended UNETR and UNETR++, and a uniform VAFT run whose inner loop is shortened until its measured GPU-hours match ours. The shortened run is the most important control. If simply running fewer attack steps on every patch matches our method at equal compute, adaptive allocation adds nothing. All methods share data splits, patch size, optimizer, epochs and a fixed tuning budget of the same number of hyperparameter trials each.

The strongest existing method is full uniform VAFT. For UNETR on Synapse it reports 75.66% clean Dice and 66.33% Dice under VAFA (Frequency Domain Adversarial Training). The file gives no reported training cost for VAFT and no reported VAFT numbers on ACDC, so we will rerun it ourselves to obtain both. Beating it means reaching robust Dice within one point of our rerun of full VAFT at no more than half its GPU-hours, or exceeding it at equal GPU-hours.

The closest transplanted method is BulletTrain ported to patches. It uses signed prediction variance averaged over each patch's voxels and its original three discrete bins, with budgets tuned to match our compute. A GAIRAT-style variant, which attacks fully but reweights by per-patch flip step count, isolates whether reweighting alone helps without compute savings.

Three ablations isolate the key ingredient. The first replaces our small-margin tail statistic with the patch-mean margin. The second replaces the model's margin with ground-truth contour distance. The third removes the moving average and uses only the current step's score. For completeness, we apply WOT, which learns a weighted average of saved checkpoints on a hold-out set, on top of both uniform VAFT and our method. WOT addresses robust overfitting rather than cost, so it should stack with our method rather than compete with it. The comparison also shows whether the hold-out data it needs is worth taking from small medical training sets.

Why not just…?

Why not just port BulletTrain to patches as-is? This is the most direct competitor, and we run it. Our expectation, which the measurement study will confirm or refute, is that a patch-averaged prediction variance is swamped by confident background voxels. It would then mark almost all patches as robust and starve the thin boundary band. If the patch-mean variant matches our tail statistic, the contribution shrinks to engineering for batches of uneven length, and we will report that plainly.

Why not just weight voxels by distance to the ground-truth contour, which costs nothing? Edge-focused designs such as HF-UNet, which adds a contour-regression branch, and the wavelet edge loss of Yu et al. improve clean segmentation. Frequency Domain Adversarial Training shows, however, that architectures without adversarial training still collapse under frequency attacks. What remains open is whether contour distance works as an allocator inside adversarial training. It might fail, because frequency attacks may flip low-contrast interiors that sit far from any contour. It might also succeed. The contour-distance ablation settles this, and we will adopt the cheaper signal if it wins.

Why not just use single-step fast adversarial training? Single-step methods are cheapest but are known to overfit catastrophically, becoming robust to single-step attacks while staying weak to multi-step ones. Nothing in the file shows this is avoided for frequency attacks on 3D segmenters. We include a single-step VAFA baseline evaluated under the multi-step attack. If it holds up, it sets the floor that any allocation scheme must beat on cost.

Why not just freeze the backbone and train robustness tokens? Robustness Tokens trains a few extra input tokens on a frozen vision transformer and reaches robust accuracy fast. On ADE20K segmentation, though, it only lifts attacked mIoU from 4.7 to 23.3, far below the 66.33% Dice full VAFT retains on Synapse. Its robustness also depends partly on the tokens staying hidden from the attacker. We consider it out of reach for clinical-grade dense robustness and cite this gap rather than rerun it.

Proposed Experiments

We use Synapse and ACDC with UNETR and UNETR++, following the attack settings of Frequency Domain Adversarial Training. We report clean Dice and 95th-percentile Hausdorff distance, Dice under VAFA and PGD, Dice restricted to a narrow band around contours, and measured GPU-hours. The main result is a curve of robust Dice against GPU-hours for uniform VAFT at several step counts, patch-level BulletTrain, and our method. The method succeeds if its curve lies above the others across the range.

The first decisive check is the measurement study. If no clean-pass statistic ranks patches by attack damage substantially better than chance, adaptive allocation cannot work, and the project becomes a negative result about dense adversarial vulnerability.

The second decisive check is the matched-compute comparison against shortened uniform VAFT. The third is the adaptive attack against starved patches. A clear robust-Dice drop there would mean the savings were bought with a blind spot, and would force a minimum per-patch budget into the design.