Opening the Spectrum Only When It Transfers: Frequency Continuation for Synthetic-to-Real Segmentation

Networks trained on rendered images, for example frames from a video game, often fail on real photographs. One well-known reason is that they learn cheap cues the renderer leaves behind. These include shader signatures, overly clean textures and noise patterns that no real camera produces. Most of these cues live in the fine detail of an image, its high spatial frequencies. Coarse layout and shape, the low frequencies, look much more alike between game and street. We propose to train the segmentation network the way coarse-to-fine optimizers solve hard inverse problems. The network first sees only the low-frequency content of both synthetic and real images. Finer frequency bands are then admitted one at a time, and each band is admitted only after a check shows it no longer separates synthetic from real. Bands that still differ are first made statistically similar before the network may use them.

The bet is that the order of training matters, not only the final data mix. In coarse-to-fine optimization, solving a blurred version of the problem first puts the optimizer in a broad basin that reflects global structure. It then refines from there instead of falling into sharp spurious minima (Total Variation Blind Deconvolution: The Devil Is in the Details, Stabilizing Diffusion Posterior Sampling by Noise--Frequency Continuation). In synthetic-to-real transfer, the "spurious minimum" is a solution built on simulator artifacts. If the network already solves the task from shape and layout before it ever sees fine detail, later fine-tuning should refine that solution rather than replace it with a shortcut.

Core Claim

A segmentation network trained with a frequency passband that widens over training will rely less on synthetic-only high-frequency cues than the same network trained on the same bands all at once. The passband widens only when each band has been checked for transferability. Lower reliance on these cues should translate into better accuracy on real images. The claim makes two testable predictions. The first is a measurable drop in sensitivity to simulator-style high-frequency patterns on real images. The second is a gain in real-domain mIoU (mean intersection-over-union across classes) over a matched-compute baseline that sees the identical final set of bands without the ordering.

Relation to Prior Work

The closest work is FDA: Fourier Domain Adaptation for Semantic Segmentation. FDA replaces a fixed low-frequency window of a synthetic image's Fourier amplitude with a real image's amplitude, and it ensembles models trained with several window sizes. It reaches 50.45 mIoU on GTA5 to Cityscapes with ResNet-101. FDA supplies our band-matching operation, and we extend it from a central square to annular bands. FDA differs from our proposal in two ways. Its split is a static preprocessing choice, and it leaves high frequencies untouched, so synthetic fine-detail artifacts stay fully visible to the network from the first iteration. We change when the network may see each band and gate that decision on measured domain separability.

The motivating evidence for treating high frequencies as the shortcut channel comes from the high-energy-physics jet-tagging literature. There, networks trained on simulation were found to exploit simulator-specific artifacts (Jet substructure classification in high-energy physics with deep neural networks, Boosted Jet Tagging with Jet-Images and Deep Neural Networks). We take this as a hypothesis to test, not a settled fact, and our diagnostic experiment checks it directly in driving scenes.

Existing coarse-to-fine approaches to domain adaptation choose different axes. Continual Coarse-to-Fine Domain Adaptation in Semantic Segmentation goes coarse-to-fine in the label taxonomy, moving from broad parent classes to 19 fine classes. It shows that sequential refinement beats direct fine-class adaptation, reaching 33.1 versus far lower mIoU in its continual setting. However, it keeps the visual input at full bandwidth throughout, so its result supports ordering in general but says nothing about ordering in frequency. Cross-domain Object Detection through Coarse-to-Fine Feature Adaptation goes coarse-to-fine from category-agnostic foreground alignment to class prototypes. It also operates on full-bandwidth features and is tied to two-stage detectors.

The method's ordering idea comes from frequency continuation in Stabilizing Diffusion Posterior Sampling by Noise--Frequency Continuation. That work restricts measurement guidance to low frequencies early and widens a radial Fourier mask later, with gains of up to 5 dB PSNR in deblurring. Its schedule is fixed in advance and tied to diffusion noise level. Ours is gated by a transferability test, which addresses the open problem of heuristic schedules named in the coarse-to-fine literature.

The coarse-to-fine literature also constrains the design. Errors committed at the coarse stage can trap later stages, as described for blind deconvolution and pose-free reconstruction (CF-NeRF). Small classes and thin structures, such as poles and signs, may simply be unsolvable from low frequencies. We therefore never discard a band. We only delay it, and we report per-class results to expose any premature-commitment damage.

Research Program

Let $\mathcal{F}$ denote the 2D Fourier transform. Partition the frequency plane into $K$ concentric annuli $B_1, \dots, B_K$ ordered from low to high. At training stage $s$, every input image $x$, synthetic or real, is passed to the network as

$$x^{(s)} = \mathcal{F}^{-1}\Big(\textstyle\sum_{k \le s} M_k \odot \mathcal{F}(x)\Big),$$

where $M_k$ is the mask for annulus $B_k$. The network therefore sees exactly the first $s$ bands, and both domains share the same bandwidth at every stage. Within each admitted band, synthetic amplitudes are matched to real amplitudes by an FDA-style swap restricted to that annulus, and phase is kept.

Moving from stage $s$ to $s+1$ requires a gate. We freeze the current backbone and fit a linear probe to tell synthetic from real images. The probe uses features computed from inputs containing bands up to $s+1$. If its held-out accuracy exceeds a threshold $\tau$ above chance, band $B_{s+1}$ still carries domain identity. In that case we strengthen its matching by replacing its full amplitude rather than blending, and we add light band-limited noise drawn from real-image statistics. Then we retest. If the band still fails after a fixed number of attempts, it is admitted with its matching at maximum strength, and the failure is logged. The final stage always uses full bandwidth, because test images arrive unfiltered.

Training at each stage uses a standard self-training recipe: supervised loss on synthetic labels plus confidence-thresholded pseudo-labels on real images. This lets the method plug into strong adaptation pipelines rather than compete with them.

We will build four things. The first is this training procedure on a DeepLab-style ResNet-101, the backbone used by FDA. The second is the band-wise domain probe. The third is a stage scheduler with a fixed total iteration budget. The fourth, and most important for the claim, is a shortcut-reliance diagnostic, described below.

Baselines

The naive reference point is a source-only model: the same network trained on GTA5 with full-bandwidth images and evaluated on Cityscapes, with no adaptation. The file contains no comparable source-only number for this backbone. The 4.5 mIoU reported by CCDA comes from its continual-step setting and is not comparable, so we will measure this baseline ourselves.

The first strong signal-processing baseline is FDA with multi-band ensembling, as described in FDA: Fourier Domain Adaptation for Semantic Segmentation. It reports 50.45 mIoU on GTA5 to Cityscapes with ResNet-101. We will reproduce it with our code and iteration budget and report both our reproduction and the published figure. It is the most direct competitor because it uses the same Fourier operation without ordering or gating.

The strongest single-source adaptation result in the file is Unsupervised Contrastive Domain Adaptation for Semantic Segmentation. It reports 60.2 mIoU on GTA5 to Cityscapes and 56.5 on SYNTHIA to Cityscapes using class-prototype contrastive alignment. Because our method changes the input schedule rather than the adaptation loss, the fair comparison is not only ours versus theirs. We will also apply our frequency continuation on top of their contrastive self-training, with identical iterations, augmentations and hyperparameter search budget. Beating it means exceeding our reproduction of their pipeline by a margin larger than the seed-to-seed spread over three seeds. Matching 60.2 alone is not the target. The multi-source result of 59.0 mIoU from Multi-Source Domain Adaptation With Collaborative Learning for Semantic Segmentation uses extra synthetic datasets and is cited only for context.

Two ablations isolate the key ingredient. The first is "same bands, no order": the network trains on the identical band-matched images from the final stage for the full budget. The second is "order without gate": the same stages run on a fixed, evenly spaced schedule, as in noise–frequency continuation. If the first ablation matches the full method, the ordering claim is false. If the second matches it, the gate is unnecessary and the method reduces to a simpler fixed schedule.

Why not just…?

Why not just blur the synthetic images, or pretrain at low resolution? Blur augmentation removes high frequencies only from synthetic data. The network still sees unmatched, full-bandwidth real images and eventually full-bandwidth synthetic images, so whatever synthetic fine detail remains is still available as a shortcut. Low-resolution pretraining is closer to our idea, because it is an ungated, unmatched form of continuation. We include it as an extra baseline: train at half and quarter resolution first, then at full resolution, with the same budget. If it equals our method, the honest conclusion is that classic resolution curricula already capture the effect. The file reports no number for this baseline on these benchmarks.

Why not just use FDA with a larger or random window? A larger window does align more of the spectrum. Our "same bands, no order" ablation is essentially that baseline extended to annular matching, so it tests this alternative directly. The file shows FDA's static approach at 50.45 mIoU, well below the 60.2 reached by contrastive alignment. This suggests spectral matching alone plateaus, but it does not show that ordering helps. Only our ablation can show that, and this is the alternative most likely to erase the gain.

Why not just use a stronger self-training adaptation method? Methods like contrastive prototype alignment already reach 60.2 mIoU, and further gains might come from better pseudo-labels alone. This is a real threat. We address it by stacking our schedule on that method rather than replacing it. If the combination yields no gain beyond seed noise, the frequency ordering adds nothing that pseudo-label quality does not already provide. The shortcut diagnostic would still tell us whether reliance on high-frequency cues differs between the two.

Why not just use domain randomization? Closing the Visual Sim-to-Real Gap with Object-Composable NeRFs reports that a domain-randomized baseline reached only 28.5 modal mask AP, against 62.7 with reconstructed real objects, in bin picking. This suggests randomization leaves systematic gaps. That evidence comes from a different task, and GTA5 assets cannot be re-randomized, so we do not run domain randomization here.

Proposed Experiments

The main benchmarks are GTA5 to Cityscapes and SYNTHIA to Cityscapes, reporting mIoU over 19 and 16 classes respectively, with three seeds each. Per-class IoU will be reported for thin structures to detect premature-commitment damage.

The decisive experiment is a planted-shortcut test. We overlay a faint, class-correlated high-frequency texture on GTA5 training pixels, invisible after low-pass filtering, and evaluate on clean Cityscapes. We also evaluate a variant where the texture appears on real images with the wrong class correlation. A model that uses the shortcut will lose accuracy on the wrong-correlation set. The core claim predicts that our method loses much less than the "same bands, no order" ablation and less than FDA.

On unmodified data, a natural-shortcut diagnostic measures the mIoU drop on Cityscapes when each annulus of a real image is replaced by the corresponding annulus from a GTA5 image. Lower sensitivity in the highest bands would directly support the mechanism.

We will also log the probe accuracy per band over training. This tests whether the gate ever fires and which bands resist matching. If no band ever exceeds $\tau$, the gate is idle and the method collapses to the fixed-schedule ablation.

As a transfer check beyond segmentation, we will run SIM10k to Cityscapes car detection. Here Cross-domain Object Detection through Coarse-to-Fine Feature Adaptation reports 43.8 AP. We would apply frequency continuation on top of a reproduction of that pipeline under a matched budget.

The proposal fails cleanly in any of three cases. The ordering ablation matches the full method on mIoU and on both shortcut diagnostics. Low-resolution pretraining matches it. Or the gain disappears when stacked on contrastive self-training.