When a large pretrained vision model is fine-tuned on synthetic images from a game engine and then used on real photographs, it often learns the simulator's quirks: its lighting, contrast and rendering style. Those cues vanish in real images, and accuracy drops. Low-rank adaptation (LoRA) trains small add-on matrices of a chosen "rank", or width, next to each frozen weight matrix. Standard practice gives every layer the same rank. We argue that under a fixed parameter budget, where the rank goes decides whether the adapter learns scene structure or simulator style. We propose to measure, for each layer, how much its usefulness to the task depends on image amplitude, the part of the Fourier spectrum that carries style and illumination, versus phase, the part that carries layout and edges. We then give rank to phase-driven layers and only a minimal rank to amplitude-driven ones.
The measurement needs no real labels. We compare each layer's gradient saliency on a synthetic image with its saliency on the same image after its low-frequency amplitude has been randomized, which changes the style while keeping the labels valid. A layer whose importance survives this style change is doing work that should transfer. A layer whose importance collapses or jumps is a likely place for shortcuts to form. The payoff is practical. VLTSeg reaches the strongest synthetic-to-real results in the file only by fully fine-tuning a 304M-parameter vision-language encoder. It also shows that simply freezing early layers hurts transfer. A rank allocator that knows which layers to starve, without freezing them, is the missing option between those two extremes.
Core Claim
The claim has two parts. First, in synthetic-to-real segmentation, the per-layer rank profile chosen by a style-invariance criterion differs from the profiles chosen by existing allocators. Second, at matched trainable parameters, that profile generalizes better to unseen real datasets than uniform LoRA or existing allocators, including when every method is given the same Fourier style augmentation. The new ingredient is an allocation criterion. It is not a new adapter architecture or a new augmentation.
Relation to Prior Work
The file contains three families of relevant work, and each plays a different role.
The first family supplies the allocation machinery. CSPLoRA plans ranks before training. It computes first-order Taylor saliency on low-confidence probe samples, smooths it logarithmically, and adds an importance-based boost on top of a minimum rank. EVA allocates rank by how much activation variance each layer explains. IGU-LoRA prunes ranks during training using integrated gradients and a signal-to-noise score. We reuse CSPLoRA's planning structure: probe once, then allocate. We change what the probe measures. All three methods score layers only by how much they matter on the training distribution. On synthetic data, "matters" includes "matters because of rendering artifacts." The file flags exactly this risk for CSPLoRA's hard-sample probing.
The second family motivates the phase/amplitude split and constrains how far it can be pushed. FDA swaps the low-frequency amplitude of synthetic images with that of real images and reaches 50.45 mIoU on GTA5→Cityscapes. This establishes that amplitude carries much of the simulator gap while phase carries semantics. FDA uses the swap only as input preprocessing and trains the full network. We use the same operation as a measuring instrument for layers. FPS finds, in MRI, that low-frequency components are the ones most susceptible to synthetic-to-real shift, and it perturbs them adaptively. This supports randomizing low-frequency amplitude as our probe. FPS relies on a heavy teacher-student network and does not address parameter efficiency.
The third family combines rank and frequency but in a different sense. FouRA learns low-rank adapters over Fourier/DCT coefficients of features, with an input-dependent gate. F-Adapter gives more adapter capacity to low-frequency shells in neural PDE operators, because that is where physical energy lies. Both allocate capacity within a layer across frequency bands. Neither tests domain shift. Our allocation is across layers, and the frequency decomposition is applied to the input as a diagnostic. F-Adapter's rule, "put capacity where the energy is," is the opposite of what we expect in sim-to-real. Here the dominant low-frequency amplitude energy is where the shortcut lives. That tension, which the file notes between F-Adapter and the high-frequency emphasis of vision losses such as Sobolev-aligned super-resolution, is something our experiments can partly resolve for this setting.
Method
We start from a frozen EVA-CLIP-style encoder with a Mask2Former decoder, following the VLTSeg recipe. Each probe image $x$ is paired with its amplitude-randomized version $\tilde{x}$. To build $\tilde{x}$, we replace the low-frequency amplitude inside an FDA-style centered window of size $\beta$ with the amplitude of another synthetic image, scaled by random gains. The phase is left untouched, so the segmentation label stays correct. No real images are used, which keeps the setting domain generalization rather than adaptation.
For each adaptable matrix $l$, we compute Taylor saliency $g_l(\cdot)$ on the segmentation loss for both views. The saliency is the magnitude of the product of the weights with their gradient, as in CSPLoRA. From this we form a style-invariance score, with expectations taken over a few hundred probe images:
$$\rho_l = 1 - \frac{\mathbb{E}\,|g_l(x) - g_l(\tilde{x})|}{\mathbb{E}\,[g_l(x) + g_l(\tilde{x})]}.$$
$\rho_l$ is near 1 when layer $l$ matters equally regardless of style. Ranks are then assigned under a total budget $R$ across $L$ matrices:
$$r_l = r_{\min} + (R - L\,r_{\min})\,\frac{\log(1 + \rho_l^{\gamma}\,\bar{g}_l)}{\sum_k \log(1 + \rho_k^{\gamma}\,\bar{g}_k)}.$$
Here $\bar{g}_l$ is the mean saliency over both views. The exponent $\gamma \ge 0$ sets how strongly invariance overrides raw importance; $\gamma = 0$ recovers a CSPLoRA-style allocation. The minimum rank $r_{\min} > 0$ ensures no layer is frozen, which respects VLTSeg's finding. The probe is a single pass costing the same order as CSPLoRA's, which the file reports at 67 seconds for an 8B language model.
We also test one optional frequency-domain term during training. It penalizes, weighted by magnitude, the phase difference between Fourier transforms of intermediate feature maps for $x$ and $\tilde{x}$. The aim is to keep the adapters from re-learning amplitude sensitivity. This term is secondary: the main claim must hold without it, and it gets its own ablation.
Research Program
The work proceeds in three stages, each with a clear chance of stopping.
The first stage is diagnostic. We compute $\rho_l$ and $\bar{g}_l$ profiles on GTA5 and SYNTHIA. We compare them, by Spearman correlation, with the rank profiles that EVA and CSPLoRA would produce. If the style-invariance profile matches the existing allocators closely, say correlation above 0.9, then the criterion adds nothing and the project ends there.
The second stage is the main comparison at matched parameter budgets. We use budgets equal to uniform rank 8, 16 and 32 across attention and feed-forward matrices.
The third stage asks whether $\rho_l$ is actually predictive rather than merely different. We use the small real validation split only for analysis, never for allocation. For each layer, we measure how much its adapter's contribution shifts real-image features, and we check whether low-$\rho_l$ layers are where synthetic-trained uniform LoRA degrades most on real data.
Baselines
The naive reference is uniform LoRA at the matched budget, trained on GTA5 with the VLTSeg recipe. It shows how much any allocation matters. We also run a frozen encoder with only the decoder trained, as a floor.
The strongest existing method is VLTSeg full fine-tuning. It reports a 63.1% mean mIoU across Cityscapes, BDD100k, Mapillary and ACDC when trained on GTA5 (VLTSeg). We reproduce it on our own hardware, with the same decoder, schedule and data, so that differences come from the adaptation method. We do not expect to beat it outright. The target is to come within about two points while training a few percent of the parameters, and to beat every parameter-efficient method at the same budget. No reported result was found on Lacuna for any LoRA-family method on the GTA5→real domain-generalization protocol, so we run all of them ourselves. FDA's 50.45 mIoU is an adaptation result that uses target images, a ResNet backbone and self-training. We cite it for context only, as it is not comparable.
The strongest parameter-efficient competitors are CSPLoRA and EVA, run on synthetic probe data with their published criteria at identical budgets. We tune each method's key hyperparameters with the same number of trials on a held-out split of the synthetic data only. Beating them means a consistent mean-mIoU gain across both source datasets, all three budgets and three seeds.
The ablation that isolates the key ingredient keeps our full pipeline and sets $\gamma = 0$. This uses the same probe images, including the amplitude-randomized views, but drops the invariance weighting. Any gain over this ablation is attributable to the criterion itself, not to the extra probe data. A second ablation adds or removes the feature phase-consistency loss.
Why not just…?
Why not just use uniform LoRA with FDA-style amplitude augmentation during training? This is the most serious threat. The augmentation may already remove most of the shortcut signal, leaving placement of rank irrelevant. We therefore give every baseline, including uniform LoRA, CSPLoRA and EVA, the identical amplitude-randomization augmentation. The claim survives only if our allocation still wins under augmentation. If the gap closes to within seed variance, we will report that input augmentation makes allocation unnecessary here. That would be a useful negative result.
Why not just run CSPLoRA or EVA on the synthetic data as they are? They do produce non-uniform allocations, and CSPLoRA's gains over uniform LoRA, such as +5.78 points on RTE, show allocation matters in language tasks. Their scores, however, reward whatever helps on the training distribution, simulator artifacts included. The stage-one correlation check and the $\gamma = 0$ ablation test directly whether that concern is real or hypothetical.
Why not just freeze the style-sensitive layers? This is our method with $r_{\min} = 0$ for low-$\rho_l$ layers. VLTSeg already reports that freezing early layers damages transfer, so we expect a minimum rank to be better than none. We include the $r_{\min} = 0$ variant to test this.
Why not just use a Fourier-domain adapter such as FouRA? FouRA changes what each adapter computes and has only been evaluated on in-domain generation and GLUE. It is complementary rather than competing. We run it as a uniform-allocation baseline at matched parameters. If it beats our allocated plain LoRA, the natural follow-up is to apply our allocation to FouRA adapters.
Proposed Experiments
The main benchmark follows the VLTSeg protocol. We train on GTA5 or on SYNTHIA and evaluate zero-shot on Cityscapes, BDD100k, Mapillary and ACDC. Metrics are per-dataset and mean mIoU, with class-wise breakdowns to see whether gains concentrate in classes defined by texture or by shape.
Transfer beyond driving is tested on 3D-IRAIS→Real-IRAIS amodal instance segmentation. There the reported full-model result is 37.10% AP for EUREKA trained only on synthetic data (IRAIS). We apply our allocation to a ViT backbone under the same protocol and report AP at matched budgets.
The decisive checks are these. First, the invariance-based rank profile must differ materially from CSPLoRA's and EVA's. Second, at every budget, it must beat the $\gamma = 0$ ablation and uniform LoRA with augmentation on mean real-domain mIoU across seeds. Third, low-$\rho_l$ layers must be the ones where uniform-LoRA features drift most on real images. If the second check fails while the first passes, the criterion finds a real structure that does not matter for accuracy, and we will say so.
References
- Strong but Simple: A Baseline for Domain Generalized Dense Perception by CLIP-Based Transfer Learning
- CSPLoRA: Confidence-Guided Structural Planning for Low-Rank Adaptation
- One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation
- IGU-LoRA: Adaptive Rank Allocation via Integrated Gradients and Uncertainty-Aware Scoring
- FDA: Fourier Domain Adaptation for Semantic Segmentation
- Bridging Synthetic-to-Real Gaps: Frequency-Aware Perturbation and Selection for Single-shot Multi-Parametric Mapping Reconstruction
- FouRA: Fourier Low-Rank Adaptation
- F-Adapter: Frequency-Adaptive Parameter-Efficient Fine-Tuning in Scientific Machine Learning
- Coloring the Noise: Adversarial Sobolev Alignment for Faithful Image Super Resolution
- Amodal Instance Segmentation with IRAIS Dataset for Sim-to-Real Transfer