Segmentation models trained on video-game frames, which come with free pixel labels, lose much of their accuracy on real street photos. The current fix with the largest gains is to make more data: re-render the game frames with an image generator so they look like rainy, nighttime, or real-camera scenes. That works, but every new target condition costs thousands of multi-step generation runs. This note proposes skipping the images entirely. Inside a frozen vision-language model (a network such as CLIP that maps images and sentences into one shared vector space), the difference between the sentence vectors for "a video game rendering of a street" and "a real photo of a street at night" already points in some direction. If that direction also describes how real night images differ from game images in the model's visual features, then adding it to the features of game frames during training is a nearly free substitute for generating a restyled dataset.
The catch, and the scientific question here, is the modality gap. Text and image embeddings in these models sit in separate, narrow cones some distance apart, so a direction measured among text vectors need not mean the same thing among image vectors. We propose to measure whether text-derived domain directions transfer into dense image features, to correct them with a simple geometric fix that uses only source images and text, and to test whether the corrected directions match expensive offline generation on synthetic-to-real generalization at a small fraction of its cost.
Core Claim
Our hypothesis is this: in the final dense feature space of a frozen vision-language encoder, the synthetic-to-real shift for a described condition is well approximated by a text-difference vector, once the component along the modality-gap direction is removed and the vector is rescaled to image-feature units. Randomly adding this corrected direction to source features, with class-specific directions applied under the source labels, should give domain-generalization gains close to those from diffusion-restyled datasets. The added training cost is a vector addition per step plus a one-time pass of the text encoder.
The optimism comes from Mind the Gap, which shows that the image–text gap behaves largely like a shift between two cones, and that moving embeddings along it changes downstream behavior in controllable ways. It reports CIFAR-10 zero-shot accuracy rising from 0.9013 to 0.9081. If the gap is mostly a constant offset, differences between text vectors cancel it, and directions may transfer even though positions do not. The worry comes from Vision Prototype Learning, which finds that text embeddings lack texture, lighting, and boundary information needed for dense prediction. A text direction may encode "night" at the level of whole-image meaning while missing the low-level changes that actually break a segmenter. The first experiment below decides which picture holds.
We write $z$ for a dense patch token from the encoder's final projected layer, which lies in the shared space. $\Delta t$ is the mean over paraphrased prompts of the target-condition text embedding minus the source-condition text embedding. $\hat g$ is the unit vector from the mean source-image token to the mean source-caption text embedding, and it estimates the gap direction. Training then uses
$$\tilde z = z + \alpha\, s\, \frac{(I - \hat g \hat g^\top)\,\Delta t}{\|(I - \hat g \hat g^\top)\,\Delta t\|},$$
where $\alpha$ is sampled from $[0, \alpha_{\max}]$ at each step and $s$ is the standard deviation of source tokens projected onto the corrected direction. The rescaling by $s$ puts the step in image-feature units instead of text-feature units. In the class-aware variant, $\Delta t$ is built per class from prompts such as "a real photo of a car" versus "a rendered car" and applied only to pixels carrying that label.
How This Sits Relative to Prior Work
Three papers use text-to-image diffusion models against synthetic-to-real shift, and each pays a different cost. DIDEX restyles GTA5 and SYNTHIA frames with ControlNet and Stable Diffusion under modular weather, location, and lighting prompts, then adapts to this pseudo-target. DIFF skips pixel generation but runs a full diffusion U-Net at every step to extract features, distilling text-conditioned features into unconditioned ones. DiDA degrades both domains with scheduled noise and adds a time-conditioned diffusion encoder, at 60–70% more GPU memory and 0.3–0.5 s per iteration. We keep DIDEX's use of text-described target conditions and drop its images. Unlike DIFF, we use no diffusion network during training. Unlike DiDA, our perturbation is semantic and chosen by text rather than by a hand-tuned noise schedule.
The VLM-era UDA study supplies our backbone and sets a constraint on the design. Swapping in an EVA02-CLIP vision encoder lifts DACS to 70.3% mIoU on GTA5→Cityscapes. The same study finds that DAFormer's ImageNet feature-distance loss runs 5–10x higher on these features, and it never uses the text encoder. We therefore build on the simple DACS-style recipe rather than on regularizer-heavy frameworks, and we put the unused text encoder to work.
Two papers treat the gap as the object of study rather than as a nuisance. MUDG shows that text queries land in empty regions of an image index and corrects this with centroids placed in text space. It is image-level only and assumes the web already covers the target. Our correction plays the same role for directions instead of positions, and it applies to dense features. Finally, SMART and DADA fuse a second modality, depth, into adaptation. They motivate the claim that cross-modal signals help synthetic-to-real transfer, but they never touch language.
Research Program
The work proceeds in three stages. Stage one is purely diagnostic and requires no training. We use EVA02-CLIP-L-14 (the backbone from the VLM-UDA study) to extract final-layer projected patch tokens for GTA5, Cityscapes, and each ACDC condition (fog, night, rain, snow). We compute "oracle" image-space directions as the difference of mean tokens between target and GTA5, both globally and per class using ground-truth masks. We then compare three text directions against these oracles by cosine similarity: raw $\Delta t$, $\Delta t$ with the gap component removed, and random directions. We also report how much of each oracle's variance the text direction explains.
Stage two trains a segmentation decoder on GTA5 over the frozen or lightly tuned encoder, using the steering rule above during training only. Test images are never modified. We sweep $\alpha_{\max}$ and the prompt sets and compare global against class-aware directions.
Stage three measures cost honestly. We report GPU-hours spent on data creation, on training, and at peak memory for our method and for every baseline, all on the same hardware.
We will also test the negative case directly: conditions whose visual signature is mostly low-level, such as sensor noise and night-time blur, may be exactly where text directions fail. VPL predicts this failure, and we report per-condition results rather than only an average.
Baselines
The naive reference is source-only training of the same decoder on the same EVA02-CLIP encoder with standard photometric augmentation, evaluated zero-shot on Cityscapes, BDD100K, Mapillary, ACDC, and Dark Zurich. The VLM-UDA study reports zero-shot gains of up to 13.7% mIoU from this backbone over older ones but no absolute per-dataset values. No reported absolute result was found on Lacuna for EVA02-CLIP source-only generalization, so we reproduce it.
The strongest existing domain-generalization method in the file is DIFF, which reports 58.01% mIoU on GTA5→Cityscapes under domain generalization, against 49.72% without its text-distillation component. DIFF uses a different backbone, so we also run DIFF-style text-conditioned distillation on our encoder where feasible. We cite the published number, not our reproduction, as the external bar.
The most important competitor is DIDEX, because it uses the same text-described target conditions and differs only in paying for images. It reports average gains of +3.8% mIoU from GTA5 and +11.8% from SYNTHIA across four targets. We re-run DIDEX-style restyling with its prompt vocabulary on the same encoder and decoder. We control for cost by running it at several generation budgets, including one matched to our total compute, and we measure its generation GPU-hours ourselves because no reported figure for DIDEX generation cost was found on Lacuna.
Two ablations isolate the key ingredients. The first removes the text, using random directions and random feature-statistic perturbation of the same magnitude in the style of SHADE. The second removes the gap correction, applying raw $\Delta t$ without projection or rescaling.
An upper reference steers with the oracle image-space direction computed from unlabeled target images. This baseline is not available in true domain generalization, but it bounds what any single-direction method can achieve.
Beating the baselines means clearing all of the following bars, averaged over the five real targets with three seeds:
- exceeding source-only and the random-direction ablation by more than seed variance;
- matching or exceeding cost-matched DIDEX;
- recovering at least half the gap between source-only and full-budget DIDEX at under one tenth of its compute.
Why not just…?
Why not just swap in the EVA02-CLIP backbone and run DACS? This is the real bar: 70.3% mIoU on GTA5→Cityscapes per the VLM-UDA study. It requires unlabeled target images, however, and gives nothing for conditions with no collected data, such as unseen night or snow domains. Our main claim is about that data-free regime. In the adaptation setting we additionally test DACS with and without steering. We expect small or no gains there, because target images already supply better directions than text does, and we will report that result as it comes out.
Why not just perturb feature statistics randomly? Random style perturbation is cheap and often strong. If text directions do no better than random directions of the same magnitude, the semantic part of the proposal is empty. The random-direction ablation settles this, per condition. Stage one also predicts the outcome before any training: a near-zero cosine between text and oracle directions would mean text adds nothing beyond randomization.
Why not just generate restyled images offline, as DIDEX does? Pixel-space restyling can alter textures and lighting that a single feature-space direction cannot express. It may therefore win outright on low-level conditions. We treat it as the accuracy ceiling for text-described generalization and compare on a cost-versus-accuracy curve rather than at one point. Dataset Diffusion also shows that generated data carries its own realism deficit: 34.2 versus 54.9 mIoU for synthetic versus real COCO. So generation is not free of the gap either.
Why not just use the modality gap as a hyperparameter and skip the correction? Mind the Gap suggests that shifting along the gap itself changes transfer, so the projection step might be unnecessary or even harmful. The raw-$\Delta t$ ablation, plus a variant that deliberately adds a gap component, tests this directly.
Decisive Checks
The project stands or falls on three results. The first is stage one's cosine between gap-corrected text directions and oracle image directions on ACDC conditions and Dark Zurich, computed against random-direction baselines. It should be clearly positive for most conditions, and correction should raise it. The second is the zero-shot generalization table across Cityscapes, BDD100K, Mapillary, ACDC, and Dark Zurich, in which steering must beat random perturbation and approach cost-matched DIDEX. The third is the cost ledger, which records generation hours, training hours, and peak memory against DIDEX, DIFF, and DiDA's reported +60–70% memory.
If stage one fails, the result is still informative, because it shows that the modality gap blocks direction transfer in dense features. In that case the proposal falls back to measuring how much target-image data the oracle direction needs, which is cheap to collect.
References
- Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
- Toward Modality Gap: Vision Prototype Learning for Weakly-supervised Semantic Segmentation with CLIP
- Generalization by Adaptation: Diffusion-Based Domain Extension for Domain-Generalized Semantic Segmentation
- Diffusion Features to Bridge Domain Gap for Semantic Segmentation
- Towards Unsupervised Domain Bridging via Image Degradation in Semantic Segmentation
- A Study on Unsupervised Domain Adaptation for Semantic Segmentation in the Era of Vision-Language Models
- Multimodal Unsupervised Domain Generalization by Retrieving Across the Modality Gap
- Energy-Based Domain-Adaptive Segmentation With Depth Guidance
- DADA: Depth-aware Domain Adaptation in Semantic Segmentation
- Dataset Diffusion: Diffusion-based Synthetic Data Generation for Pixel-Level Semantic Segmentation