Segmentation models trained on video-game driving scenes (GTAV, SYNTHIA) lose much of their accuracy on real streets. A simple remedy works well. Split each training image into the parts of its Fourier transform that carry layout and the parts that carry appearance, keep the layout parts fixed, and randomly rescale the appearance parts. Because the pixel layout never moves, the label masks stay valid, and the model learns to ignore surface statistics it should not trust. The current version of this remedy applies one perturbation profile to the whole image. A red traffic light and a grey asphalt road therefore receive the same color and texture jitter. Yet work on domain-specific augmentation shows that jittering a cue the label depends on, such as hue for a traffic light, teaches the model to ignore real evidence.
We propose to make spectral augmentation class-conditional, and to derive each class's perturbation budget from data instead of expert rules. The model trains on two synthetic sources. For each class, each frequency band and each color channel, we measure how much the two sources disagree, relative to how much that class varies within a source. Where the sources disagree, the cue is probably rendering-specific nuisance, so we perturb it heavily for that class. Where they agree, the cue probably carries meaning, so we leave it nearly untouched. The augmented class regions are recombined using the ground-truth masks, so no label is corrupted.
Core Claim
Synthetic-to-real segmentation gains from Fourier augmentation are limited because the augmentation is spatially uniform, not because it is too weak. A per-class, per-band perturbation budget estimated only from source data should improve zero-shot mean intersection-over-union (mIoU) over a uniform spectral augmentation tuned with the same budget. The gain should concentrate on small, color-defined classes such as traffic lights and signs, and should not cost accuracy on large "stuff" classes. The new element is a source-only, data-derived rule for which appearance statistics may vary per class. The class-conditional compositing itself is close to engineering. The estimation rule is what we test.
Where This Sits
The method we build on is PASTA. It keeps the Fourier phase (roughly, where edges and shapes sit) fixed and multiplies the amplitude spectrum (roughly, how much energy each frequency carries, which governs texture, contrast and color statistics) by random factors that grow with frequency. It raises a DeepLabv3+ from 28.95 to 44.12 mIoU on GTAV→Cityscapes, but it applies one profile to every pixel. SHADE varies style through per-channel feature means and variances. It is also class-agnostic and ignores high-frequency texture. FIESTA comes closest to spatial selectivity: it uses model uncertainty to decide where to blend Fourier-augmented views in medical scans. Its signal is "where the model is unsure," not "which cue is nuisance for this class," and it was never run on multi-class driving scenes. CamoFA swaps low-frequency amplitude while keeping high-frequency amplitude and phase, with a single global cut-off found by Bayesian optimization, and is evaluated only in-domain.
The motivation comes from the domain-constraint literature. Heavy color distortion prevents separating subclasses that differ by color (MaskCon). Photometric augmentation destroys the tissue-versus-tool chromatic cues in surgical video (Dissecting self-supervised learning methods for surgical computer vision). Photometric jitter corrupts terrain spectral signatures (SwinLS). All three encode their constraints by hand, at the level of a whole dataset. We replace the hand rule with a measurement and move the constraint from the dataset to the class.
Two papers constrain the design. VIPAug shows that phase is not truly invariant under blur and defocus, so keeping phase fixed leaves one failure mode untreated. We treat that as out of scope and add standard blur as a shared augmentation for all methods. Augmentation-based Domain Generalization for Semantic Segmentation shows that well-chosen rule-based augmentations alone reach 39.5 mIoU (DeepLabV2) and 44.2 mIoU (DAFormer) on SYNTHIA→Cityscapes. Any spectral method must therefore be measured on top of a strong rule-based pipeline, not against a bare model.
Research Program
The setting is multi-source zero-shot generalization. We train on GTAV and SYNTHIA with no target images, and select checkpoints only on held-out source images. This avoids the target-validation leakage that the file flags as common in domain generalization papers.
The first component is the budget estimator. We split each source image into its ground-truth class regions. For each region we compute the log-amplitude spectrum of the masked, windowed region, pooled into $B$ radial frequency bands per color channel. For class $c$, band $b$ and channel $k$ we form a separation score:
$$ r_{c,b,k} = \frac{(\mu^{G}_{c,b,k} - \mu^{S}_{c,b,k})^2}{\sigma^{2}_{c,b,k}}. $$
Here $\mu^{G}$ and $\mu^{S}$ are the mean pooled log-amplitudes of that class-band-channel in GTAV and SYNTHIA, and $\sigma^2$ is the pooled within-source variance. A large $r$ means the two renderers disagree about this statistic for this class, so it is treated as nuisance. A small $r$ means the statistic is stable across sources and is protected. The per-class perturbation scale is a monotone map of $r$, clipped at both ends. On top of it we keep PASTA's frequency-increasing profile as a floor. The floor matters because both sources are synthetic and share a deficit of high-frequency sensor noise that cross-source disagreement cannot reveal.
The second component is the augmentation operator. We group classes into four clusters by their budget vectors, which keeps the cost at four FFT passes per image. Each cluster's profile is applied to the whole image with phase fixed, and the outputs are composited using the ground-truth masks. The masks are feathered over a few pixels so that compositing does not create fake edges. Labels are untouched by construction.
The third component is analysis. We check whether the estimated budgets match intuition, for example whether traffic-light hue bands receive low budgets and road texture bands high ones. We also check whether per-class IoU gains track the budgets.
Baselines
The naive reference is DeepLabv3+ (ResNet-101), and separately DAFormer, trained on GTAV+SYNTHIA with the rule-based pipeline found by Schwonberg et al. (Augmentation-based Domain Generalization for Semantic Segmentation). It is the "do nothing spectral" point, and it already includes cropping, elastic transforms and blur.
The strongest existing competitor is global PASTA (PASTA) on top of the same pipeline. It is the closest method that preserves structure, and it reports 44.12 mIoU single-source GTAV→Cityscapes with DeepLabv3+. We found no reported result on Lacuna for PASTA in the GTAV+SYNTHIA multi-source setting, so we rerun it ourselves.
The second competitor is SHADE (SHADE), which reports 45.11 mIoU averaged over Cityscapes, BDD100K and Mapillary in exactly this multi-source setting. That is the headline number to beat on the three-target average. Its backbone and protocol may differ from ours, so we also rerun it under our protocol.
Fairness rules apply to all methods. They use identical data, schedules and seeds (three per method), and the same total hyperparameter search budget. PASTA's global strengths get the same number of trials as our clip bounds. Model selection is on source validation only.
Three ablations isolate the key ingredient. The first, random per-class profiles, assigns budgets drawn from the same marginal distribution but shuffled across classes. It tests whether the estimated budgets matter or whether per-class diversity alone helps. The second, binary protection, applies global PASTA and then pastes back original pixels for a hand-picked set of color-defined classes. The third, the no-floor variant, tests whether cross-source disagreement alone misses the shared synthetic deficit.
Beating PASTA means a mean three-target mIoU gain larger than the seed spread, with no target dataset worse. Beating SHADE means exceeding its rerun three-target average under our protocol.
Why not just…?
Why not just tune PASTA's global strength more carefully? This is the main threat. A weaker global jitter could protect color-defined classes almost as well while still regularizing everything else. Equal tuning budgets settle the question. We also report per-class IoU. A better-tuned global PASTA must trade small-class accuracy against large-class accuracy along a single knob. If our method beats the best point on that trade-off curve, class conditioning matters. If it does not, the claim fails.
Why not just hand-specify the constraints, protecting hue for lights and signs? This is the binary-protection ablation, and it is how the domain-constraint literature works today (MaskCon, SwinLS). If hand rules match the estimator, the honest conclusion is that the estimator is a convenience, not a gain. We would still report whether it recovers the expert rules without being told them, which matters for domains where no expert rule exists.
Why not just use feature-level style hallucination as in SHADE? SHADE changes only channel means and variances. By its own design it ignores high-frequency texture, the gap PASTA identifies as dominant in synthetic-to-real shift. The two approaches are not exclusive, so we also test SHADE plus our augmentation.
Why not just use diffusion-generated restyled images as in CDGA? CDGA gains on classification (PACS average 64.3 to 68.4). However, diffusion resynthesis moves boundaries and hallucinates objects, which breaks pixel masks. We do not run it as a main baseline. Making it mask-faithful is a separate research problem.
Decisive Checks
We evaluate on Cityscapes, BDD100K and Mapillary Vistas with mIoU, per-class IoU, and boundary-band IoU (IoU restricted to pixels near label edges). Boundary-band IoU detects any damage from mask compositing.
Three outcomes would confirm the claim. First, our method beats global PASTA on the three-target average under equal tuning. Second, the gain concentrates on small, color-defined classes, and per-class gain correlates positively with protection. Third, the random-profile ablation recovers little of the gain.
Three outcomes would refute it. Random profiles matching estimated ones would mean the estimator is decorative. Global PASTA at its best strength matching our method would mean conditioning adds nothing beyond tuning. A drop in boundary-band IoU would mean compositing introduces artifacts that cancel the benefit.
We additionally test generalization of the estimator itself. We repeat the analysis on Sim10K→Cityscapes detection, where PASTA reports +16.9 mAP over its baseline, to see whether class-conditional budgets transfer to box-level tasks.
References
- Pasta: Proportional Amplitude Spectrum Training Augmentation for Syn-to-Real Domain Generalization
- Style-Hallucinated Dual Consistency Learning for Domain Generalized Semantic Segmentation
- FIESTA: Fourier-Based Semantic Augmentation With Uncertainty Guidance for Enhanced Domain Generalizability in Medical Image Segmentation
- CamoFA: A Learnable Fourier-Based Augmentation for Camouflage Segmentation
- MaskCon: Masked Contrastive Learning for Coarse-Labelled Dataset
- Dissecting self-supervised learning methods for surgical computer vision
- SwinLS: Adapting Swin Transformer to Landslide Detection
- Domain Generalization with Vital Phase Augmentation
- Augmentation-based Domain Generalization for Semantic Segmentation
- Leveraging Generative Foundation Models for Domain Generalization