Where Style Enters a Hierarchical Encoder: Stage-Targeted Spectral Perturbation for Synthetic-to-Real Segmentation

Segmentation models trained on video-game images (GTA5, SYNTHIA) lose much of their accuracy on real street scenes. The strongest fixes available today attach a 300M-parameter foundation model to the problem, which is too slow for a car or a robot. Lightweight hierarchical vision transformers are a different family: they build a feature pyramid at 1/4, 1/8, 1/16 and 1/32 of the input resolution, like a convolutional network, and some of them run in real time. This proposal asks a narrow question about them. Inside such a pyramid, at which stage does source-specific "style" (synthetic lighting, texture, color rendering) get absorbed? And does disrupting style only at that stage, during training only, generalize better than disrupting it everywhere or only at the input?

The mechanism works on the frequency spectrum of each feature map, which splits into an amplitude part and a phase part. Prior work links amplitude to appearance and phase to shape. The core idea is to randomize feature amplitude while keeping phase, at stage-specific strengths chosen without any real-domain data, and to require the model's prediction to stay the same under that randomization. The perturbation is removed at test time, so the deployed network is an unmodified SegFormer with no added parameters or latency. The claim can be tested in two sentences' worth of experiments. Early high-resolution stages should need strong amplitude randomization and late stages little. Placing the perturbation this way should beat both input-level spectral augmentation and a uniform per-stage perturbation.

Core Claim

In an encoder pre-trained on ImageNet-1k and fully fine-tuned on synthetic data, source-specific appearance is concentrated in the amplitude spectra of the first two pyramid stages. Two things follow. A training-only, phase-preserving amplitude perturbation, weighted toward those stages and paired with a prediction-consistency loss, should raise synthetic-to-real mIoU over the strongest augmentation-only recipe for the same architecture. It should also beat an equally strong perturbation applied at the input or spread uniformly over stages. What is new is not the phase/amplitude split, which Learning Spectral-Decomposed Tokens for Domain Generalized Semantic Segmentation already uses. The new parts are three: a stage-resolved account of where style lives in a native pyramid, an intervention that follows that account, and zero inference cost.

Positioning Against Prior Work

The closest work is SET. It freezes a flat DINOv2 backbone and inserts learnable tokens (6–9M parameters) that process the phase and amplitude of intermediate features separately. It reports a 1.66 mIoU gain over the previous best on GTA5→Cityscapes and +3.12 on ACDC snow. SET motivates our proposal, because it shows that the amplitude/phase split carries domain information inside transformer features. It differs from ours in three ways. It runs on a single-scale foundation model, its tokens remain at inference, and it applies the same treatment at every depth. We use a multi-stage encoder, perturb only during training, and treat the stage-by-stage allocation as the object of study.

Domain Generalization for Semantic Segmentation: A Survey supplies the premise that style sits mostly in amplitude and structure in phase. It also sets the gap we target: foundation-model systems (SoRA at 68.3% mIoU, MFuser above 71% on GTA5→Cityscapes) cannot meet real-time budgets, and they rely on web-scale pre-training. The survey calls that reliance the "proxy data paradox."

The architecture comes from SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers. SegFormer uses overlapped patch merging and has no explicit positional embeddings. These properties matter for us. Amplitude perturbation of a stage's features does not collide with a learned position table, and the all-MLP decoder reads every stage directly, so a perturbation at any stage reaches the loss.

Augmentation-based Domain Generalization for Semantic Segmentation plays two roles. It constrains the design and it is the main competitor. It shows that well-chosen rule-based augmentations, chiefly elastic transform and random resized crop, carry DAFormer to 44.2% mIoU on Synthia→Cityscapes/BDD. It also finds negative interactions between similar distortions, which warns that stacking our perturbation on top of input-space style augmentation may not add up. That paper selected checkpoints on target data; we do not.

Two classification papers suggest that intermediate transformer layers are where shortcuts form. Self-Distilled Vision Transformer for Domain Generalization fixes this by distilling the final prediction into intermediate blocks, and Gradient Starvation: A Learning Proclivity in Neural Networks does so by penalizing logit norms. Both motivate a stage-resolved view, but neither was built for dense outputs or pyramid encoders.

VLTSeg: Simple Transfer of CLIP-Based Vision-Language Representations for Domain Generalized Semantic Segmentation marks the accuracy ceiling. It gives a reference point, not a like-for-like comparison.

Research Program

The first part is diagnostic. We train SegFormer with MiT-B2 and MiT-B5 encoders on GTA5 and on SYNTHIA using the standard recipe. At each stage $s$ we compute per-channel 2D Fourier transforms of the feature maps and run an amplitude-swap test. For a synthetic image, we replace the stage-$s$ feature amplitude with the amplitude from a different domain's image, keep the phase, and measure how much the output segmentation changes and whether mIoU on real images improves.

The swap test runs twice. For method design it uses a second synthetic domain: SYNTHIA amplitudes injected into a GTA5-trained model, which keeps every decision source-only. For post-hoc analysis only, it uses Cityscapes amplitudes. The prediction is that swaps at stages 1–2 move real-domain accuracy most and swaps at stages 3–4 mostly damage it. If instead late stages dominate, the premise that style is a high-resolution phenomenon in pyramids is wrong, and we report that.

The second part is the intervention. For a feature map $z_s$ at stage $s$, write its channel-wise Fourier transform as amplitude $A_s$ and phase $P_s$. We form

$$\tilde z_s = \mathcal{F}^{-1}\big(\left[(1-\lambda_s)A_s + \lambda_s A'_s\right] e^{iP_s}\big),$$

where $A'_s$ is the stage-$s$ amplitude of another image in the batch, and $\lambda_s \sim U(0,\alpha_s)$ is a random mixing weight. Its per-stage ceiling $\alpha_s$ is the only new hyperparameter per stage. The perturbed forward pass continues from $\tilde z_s$.

Training minimizes

$$\mathcal{L} = \mathcal{L}_{CE}(\hat y) + \mathcal{L}_{CE}(\tilde y) + \beta\,\mathrm{KL}(\hat y \,\|\, \tilde y),$$

where $\hat y$ and $\tilde y$ are the per-pixel class predictions from the clean and perturbed passes, and $\beta$ weights their agreement. Because amplitude is mixed within the batch and phase is untouched, spatial layout, and thus the label map, is preserved exactly. This avoids the label-geometry breakage the file flags for aggressive spatial augmentation in dense tasks.

The third part is source-only selection. The $\alpha_s$ values, $\beta$ and the checkpoint are chosen by mIoU on the other synthetic dataset: SYNTHIA when training on GTA5, and GTA5's overlapping classes when training on SYNTHIA. Real targets are touched only once, at final evaluation. We separately report how well synthetic-proxy selection agrees with the oracle target-selected choice. That agreement is itself a result on the open source-only selection problem.

Baselines

The do-nothing reference is SegFormer (MiT-B2 and MiT-B5, ImageNet-1k pre-training) trained on the source with its default augmentation. We know of no reported result on Lacuna for SegFormer/MiT trained on GTA5 or SYNTHIA and evaluated on real targets, so we measure it ourselves.

The strongest competitor at matched scale is the augmentation recipe of Augmentation-based Domain Generalization for Semantic Segmentation, with elastic transform and random resized crop at the top of its factorial design. It reports 44.2% mIoU on Synthia→Cityscapes/BDD100K with DAFormer, whose encoder is a MiT, so the architecture is directly comparable. We rerun that recipe on our exact encoder, schedule and seeds. Its hyperparameters are re-selected with the same synthetic-proxy protocol, which removes the target-selection advantage the original number enjoyed. Beating it means an average gain of at least 1.5 mIoU across Cityscapes, BDD100K and Mapillary over three seeds, with non-overlapping seed spreads. Our method is also stacked on top of this recipe, since a practitioner would use both.

Input-level amplitude mixing is the same operator applied to the RGB image rather than to features. It is the cheapest spectral alternative and uses the same tuning budget.

Uniform feature amplitude mixing sets one shared $\alpha$ at all stages. This ablation isolates the stage-allocation claim. Single-stage variants that perturb only stage 1, 2, 3 or 4 map the dependence directly.

Channel-statistics perturbation replaces the Fourier step with mixing each channel's mean and standard deviation between images at the same stages. It tests whether the full spectrum matters, or only its zero-frequency component and overall energy.

Spectral Decoupling from Gradient Starvation: A Learning Proclivity in Neural Networks, applied as a per-pixel logit-norm penalty, represents the shortcut-regularization line.

All baselines get the same data, iteration count, crop size and a tuning grid of equal size, all selected by the synthetic proxy.

As an upper reference, not a competitor, we quote VLTSeg's 63.1% mean mIoU over four real targets from GTA5 with EVA-02-L-14 (about 304M parameters), together with SoRA's 68.3% and MFuser's above 71% on GTA5→Cityscapes from the survey. We report latency next to accuracy, so that the gap is stated in both mIoU and frames per second.

Why not just…?

Why not just use stronger input augmentation, including Fourier amplitude mixing on the image? This is the most serious threat. The factorial study shows augmentation alone reaches 44.2% with a MiT-based model. Input-level amplitude mixing perturbs all stages indirectly, through whatever the first layers pass on. Our diagnosis predicts that style is re-expressed inside the stage-1 and stage-2 features after overlapped patch merging and Mix-FFN, where input perturbation has only partly reached. If input-level mixing matches stage-targeted feature mixing within seed noise, the feature-level mechanism is unnecessary. We would then report the stage diagnosis as the contribution and recommend the input version.

Why not just perturb channel means and variances at each stage, which is cheaper than a Fourier transform? This is also a real threat, because channel statistics are the zero-frequency component and the total energy of the amplitude spectrum. The channel-statistics baseline at identical stage allocation settles it. If it ties, the claim shrinks to "where," not "how," and the lighter operator wins. We expect the full spectrum to help mainly on texture-heavy classes (road, vegetation, terrain) and on the ACDC snow and fog conditions where SET saw its gains. Per-class and per-condition breakdowns will test that expectation specifically.

Why not just fine-tune a foundation model as VLTSeg does? Because the deployment question is different. VLTSeg concedes that its 304M-parameter backbone is hard to run on edge devices, while SegFormer-B0 runs at 48 frames per second on Cityscapes. That backbone's generalization also comes partly from web-scale pre-training, the proxy-data paradox the survey describes. We do not claim to match 63.1%. We claim to move a real-time pyramid encoder, trained with ImageNet-1k only, measurably closer to it.

Why not just add SET's spectral tokens to MiT? It is possible, but those tokens stay at inference and were designed for a frozen backbone. We include a SET-style variant on a frozen MiT as a secondary comparison. If it outperforms our training-only perturbation on a fully fine-tuned MiT, that result would bear on the full-fine-tuning versus frozen-backbone tension between VLTSeg and SET, and we would report it either way.

Proposed Experiments and Decisive Checks

The sources are GTA5 and SYNTHIA. The targets are Cityscapes, BDD100K and Mapillary Vistas, reported per target and as a mean. For real-to-real robustness, models trained on Cityscapes are evaluated on ACDC by condition and on Cityscapes-C by corruption type and severity. Encoders are MiT-B2 and MiT-B5, with MiT-B0 added to check that the effect survives at the real-time end. Metrics are mIoU over three seeds with standard deviation, per-class IoU for texture-dominated versus shape-dominated classes, and inference frames per second, which should be identical to plain SegFormer.

Four outcomes are decisive.

  1. The amplitude-swap diagnosis on the synthetic proxy should identify stages 1–2 as the main carriers of domain appearance, and the post-hoc real-target analysis should agree.
  2. Stage-targeted feature perturbation should beat both input-level amplitude mixing and uniform-stage mixing by at least 1.5 mean mIoU at matched tuning budget.
  3. Full-spectrum mixing should beat channel-statistics mixing at the same stages, or the claim narrows as described above.
  4. The hyperparameters selected on the synthetic proxy should land within 1 mIoU of the oracle target-selected setting. That would make the source-only protocol credible for future domain-generalization work on this benchmark family.

A failure of the first check would falsify the core premise and redirect the work toward late-stage mechanisms, such as the stage-aware self-distillation suggested by SDViT.