Models trained on video-game imagery such as GTA5 still lose a lot of accuracy on real street photos. Real images have lens blur, sensor noise and messy textures. Synthetic images have crisp shader edges and clean surfaces. Much of this difference is a difference in frequency content: how much image energy sits in coarse, slowly varying patterns versus fine, rapidly varying detail. We propose to insert small filters inside a segmentation network. Each filter reweights the frequencies of intermediate features, and it is trained with an explicit signal that says which frequency bands look different between synthetic and real images. Bands that differ across domains get turned down. Bands that agree across domains get turned up. A fixed energy budget stops the filter from deleting information.
The two-sentence version is this. Earlier Fourier methods for sim-to-real either swap fixed low-frequency image statistics at the input, or learn frequency filters without any knowledge of the domain gap. We learn in-network, continuous frequency masks whose training objective directly penalizes the synthetic-versus-real spectral mismatch in each band, under an energy-preservation constraint. This targets the fine structures (poles, signs, riders) where synthetic training fails and where a blunt blur would destroy the signal.
Core Claim
In a Transformer-based unsupervised domain adaptation (UDA) pipeline, the synthetic-to-real gap in intermediate features should be concentrated in identifiable radial frequency bands. UDA here means training on labeled synthetic images plus unlabeled real ones. A gate that is told where the gap is, rather than one learned from the segmentation loss alone, should suppress simulator shortcuts while keeping thin-structure detail. The claim is falsifiable at two points. The first is a measurement step that may show no concentrated feature-level gap. The second is an ablation that trains the identical gate without the discrepancy signal.
Relation to Prior Work
Three works supply ingredients, and none combines them.
FDA swaps a fixed central window of the synthetic image's Fourier amplitude with a real image's amplitude, using one hyperparameter β. It motivates our premise that the spectrum carries the domain gap. Our design differs in three ways: it acts on internal features rather than RGB input, it is learned rather than fixed, and it is not limited to low frequencies.
When Semantic Segmentation Meets Frequency Aliasing contributes two modules. One is a parameter-free filter that removes aliasing, meaning frequencies above what a downsampling step can represent. The other is FreqMix, a learned attention module that reweights frequencies above and below that limit. Both were evaluated only in-domain, on Cityscapes, PASCAL VOC and ADE20K, mostly with CNNs. We reuse its machinery for layer-wise spectral analysis. The key difference is that FreqMix has no cross-domain signal, so on synthetic data nothing stops it from amplifying exactly the shortcut frequencies that help on the source.
FAMNet splits deep features into three hand-set bands for few-shot CT-to-MRI segmentation and favors the mid band. It reports that uniform attention across bands costs over 7% Dice. That result motivates band selectivity. FAMNet's reported degradation on small targets constrains us: its fixed bands are too coarse for urban scenes, so we learn a continuous radial mask instead.
Beyond Attention Imbalance (FLASH) makes two points we build on. It argues that multi-head attention behaves as a low-pass filter. It also stabilizes a training-free spectral edit by conserving total energy, which is Parseval's theorem: signal energy is the same whether computed in space or in frequency. We borrow the energy constraint as our anti-collapse device.
FLASH's low-pass finding seems to cut against us, since attention may already remove synthetic high-frequency shortcuts on its own. We address this directly. Uniform low-pass filtering removes shortcut detail and thin-object detail together, and our measurement step tests whether a selective gap survives it.
DAFormer is our host system and strongest baseline, using its code at lhoyer/DAFormer. It combines a Mix Transformer encoder, rare-class sampling and ImageNet feature distillation, and works entirely in the spatial domain. Unsupervised Contrastive Domain Adaptation aligns class prototypes spatially and represents the spatial-alignment alternative.
Research Program
The first step is measurement, before any module is built. We train DAFormer on GTA5 → Cityscapes. At every encoder stage (patch embedding, each block, each downsampling step) we compute, per channel group, the radially averaged log-amplitude spectrum of the features for synthetic and real batches. We then plot their gap as a function of radial frequency. We also compute the aliasing score from Chen et al. at the patch-embedding steps.
This step has two possible outcomes. The gap might sit in a few specific bands, for example mid-high bands right after patchification, where sharp synthetic edges alias. In that case a selective gate has something to act on. If instead the gap is diffuse, or confined to low frequencies at the input, then FDA-style input swapping should already be enough and we would report that as a finding.
The second step is the gate. At a small number of chosen stages, a feature map $z$ is filtered as
$$\tilde z = \mathcal{F}^{-1}\big(m(r)\odot \mathcal{F}(z)\big), \qquad \sum_r m(r)^2 E_z(r) = \sum_r E_z(r),$$
where the symbols are defined as follows. $\mathcal{F}$ is the 2D FFT. $r$ indexes radial frequency bins. $m(r)\ge 0$ is a smooth mask, parameterized by a few radial basis functions per channel group. $E_z(r)$ is the feature energy in bin $r$. The constraint keeps total energy fixed, so turning one band down forces others up.
The mask is trained with the usual DAFormer losses, supervised on source and self-training on target, plus a discrepancy term:
$$\mathcal{L}_{spec} = \sum_r m(r)^2\,\big|\log \bar A_s(r) - \log \bar A_t(r)\big|.$$
Here $\bar A_s$ and $\bar A_t$ are running averages of the synthetic and real feature amplitudes in bin $r$. They are detached from the gradient, so the loss moves the mask and never the features. The energy constraint shifts weight toward bands where the domains agree, and the task loss pushes back wherever a discrepant band still carries label information.
We compare two versions of the mask: a global learned mask, and a mask predicted per image from its own radial spectrum by a tiny network.
The third step is analysis. We check whether the learned masks rediscover FAMNet's mid-band preference or something different. We check whether the masks vary across stages. We check whether the residual spectral gap after gating correlates with per-class IoU gains on thin classes.
Baselines
The naive reference point is DAFormer's source-only model trained on GTA5 and tested on Cityscapes with no adaptation. This bounds how much any adaptation helps.
The strongest competitor is full DAFormer, which reports 68.3 mIoU on GTA5 → Cityscapes (DAFormer). It is the strongest because it already exploits Transformer shape bias, the property our proposal claims is incomplete. We rerun it from the official code with three seeds, the same schedule, the same GPU budget (roughly the reported 16 hours on one RTX 2080 Ti per run) and the same crops, because UDA results vary noticeably between seeds.
On SYNTHIA → Cityscapes, the file gives only DAFormer's +5.4 mIoU gain over prior work. No reported absolute DAFormer SYNTHIA result was found on Lacuna, so our rerun provides that reference. The contrastive method reports 56.5% there and 60.2% on GTA5 (Unsupervised Contrastive DA). No reported per-class DAFormer IoU was found on Lacuna for pole, traffic sign, rider or bicycle. The file reports only train (65.1) and truck (74.5), so the thin-class comparisons come from our own reruns.
Beating DAFormer means a mean mIoU gain larger than the seed spread on both benchmarks, with the gain concentrated on thin and rare classes rather than on road or sky.
The ablation that isolates the key ingredient is the identical gate, with the same stages, parameters and energy constraint, trained without $\mathcal{L}_{spec}$. We also run a FreqMix-style variant from Chen et al. inserted at the same stages. If either matches the full method, the discrepancy signal is not doing the work.
We add two further variants under the same tuning budget. One adds FDA input swapping on top of DAFormer, with β tuned on a held-out split of source images. The other replaces our mask with a static feature-level amplitude swap at the same stages.
Why not just…?
Why not just add FDA's input-level amplitude swap to DAFormer? It is free and already known to help: FDA alone reaches 50.45 mIoU on GTA5 → Cityscapes with an older backbone (FDA). Its limitation is that it only changes low-frequency image statistics, and it cannot touch gaps created inside the network, such as aliasing at patch embedding. If our measurement step finds the feature-level gap is mostly inherited from the input's low frequencies, this baseline should tie us. We run it at an equal tuning budget and treat a tie as a real negative result.
Why not just plug in FreqMix or the de-aliasing filter? This is the most serious threat. Part of any gain we see could simply come from anti-aliasing, which already gives +1.6 mIoU in-domain on Cityscapes (Chen et al.). Our answer is the matched ablation described under Baselines. If a FreqMix gate trained without the domain signal performs as well, the right conclusion is "anti-aliasing helps UDA," not our claim. The discrepancy signal should also show a qualitative difference: on source data it should suppress bands that FreqMix amplifies.
Why not just blur or add noise to synthetic images? Chen et al. report that Gaussian blur weakens all high frequencies and erases the boundaries that thin classes depend on. Blur and noise also do not adapt to the real target. We include blur plus noise augmentation, tuned on held-out source images, as a baseline, and compare per-class IoU on poles and signs, where indiscriminate blur should hurt most.
Why not just align features spatially with contrastive prototypes? Spatial alignment and spectral gating are not exclusive. The file's evidence that frequency features transfer better than spatial ones comes from medical few-shot data (FAMNet), not from street scenes. So we do not claim superiority in advance. We report our gate both alone and combined with DAFormer's existing training, and compare against the contrastive method's reported numbers.
Decisive Checks
The work stands or falls on three results.
The first is the measurement step. It must find a feature-level spectral gap concentrated in bands the input swap does not reach.
The second is the matched ablation. The discrepancy-trained gate must beat the same gate trained without $\mathcal{L}_{spec}$, and beat FreqMix at the same stages, by more than the seed spread across three seeds on both GTA5 and SYNTHIA → Cityscapes.
The third is where the gain comes from. The mIoU gain over our DAFormer rerun must come mainly from thin and rare classes, and it must track reductions in the residual band-wise gap.
We also report the extra FFT latency per image, since transform cost is a known limitation of frequency methods. A gain that disappears at matched compute will be stated as such.
References
- FDA: Fourier Domain Adaptation for Semantic Segmentation
- When Semantic Segmentation Meets Frequency Aliasing
- FAMNet: Frequency-aware Matching Network for Cross-domain Few-shot Medical Image Segmentation
- Beyond Attention Imbalance: Mitigating Hallucinations via Spectral Surgery
- DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation
- Unsupervised Contrastive Domain Adaptation for Semantic Segmentation
- Playing for Data: Ground Truth from Computer Games
- The Cityscapes Dataset for Semantic Urban Scene Understanding
- The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes