A Few Generated Images Instead of Thousands: Exemplar-Anchored Feature Synthesis for Zero-Shot Adaptation

A segmentation model trained on clear-weather driving scenes fails at night or in snow, and in the zero-shot setting we have no real target images to fix this. Two families of methods attack the problem. The first renders every training image into the target condition with a diffusion model, a generator that turns random noise into images by gradually denoising it. This works well, but it costs more than a minute per image. The second never renders pixels. It nudges the network's internal feature statistics toward a text description such as "driving at night" and costs about a third of a second per image, but it falls behind. Our claim is that these two families sit at the ends of one generation-budget axis, and that the useful operating point lies in between. Generate a handful of target-condition images once, never paired with or aligned to the source data. Use them only to measure what the target domain looks like inside the network's visual feature space. Then synthesize unlimited restyled source features from those measurements, trained with a class-aware contrastive loss that uses the exact source labels.

The reason this should work is geometric. Text embeddings and image embeddings in vision-language models like CLIP occupy separate regions of the shared space, a phenomenon called the modality gap. A text description of "night" is therefore a misplaced target for visual features. Per-image diffusion, meanwhile, spends most of its compute re-rendering scene content that the source image already provides. A small set of generated exemplars supplies the visual-space target that text cannot. Transferring only their feature statistics, rather than their pixels, avoids both the per-image cost and the risk that the generator changes the scene layout and invalidates the ground-truth masks.

Core Claim

On Cityscapes→ACDC zero-shot adaptation, we expect segmentation accuracy as a function of the number of generated images $K$ to saturate early. Specifically, we expect $K$ on the order of tens of unpaired generated images per condition, used as anchors for visual feature statistics, to recover most of the gap between text-driven feature steering and per-image diffusion synthesis. The total generation compute would be one to two orders of magnitude lower than per-image synthesis. We further claim that part of text steering's shortfall comes specifically from the modality gap rather than from text being uninformative. A gap-corrected text target should therefore recover some, but not all, of the difference. The remainder should be attributable to spatially non-uniform effects that only visual exemplars capture.

Where This Sits

Three papers motivate the proposal by establishing the two ends of the budget axis. PØDA optimizes channel means and standard deviations of an early backbone layer so that pooled features match a CLIP text embedding of the target prompt. It reports roughly 0.3 seconds per augmentation versus over a minute to generate a synthetic image. ZoDi and SIDA instead generate a target-style image for every source image. ZoDi uses layout-conditioned diffusion with stochastic inversion. SIDA uses a chain of GPT-4o captions, Stable Diffusion, and image-to-image translation, and argues explicitly that text cannot express fog density or wet-road glare. Both concede that the generator can alter scene geometry, which corrupts the masks. We keep PØDA's cheap feature-level restyling but replace its text target with visual-space statistics. We keep SIDA's insight that images carry spatial detail text lacks, but we spend generation on $K$ images rather than one per training image.

Two papers explain why the text target is misplaced. Mind the Gap shows that CLIP's image and text embeddings sit in separate cones, a separation caused by initialization and maintained by the contrastive loss, and that shifting the gap changes downstream accuracy. VPL shows that for dense segmentation, text prototypes lack texture and lighting detail and should be translated into the visual embedding space. VPL does this only in-domain. We carry VPL's conclusion into the domain-shift setting, where the target domain itself, not just the class, must be represented visually.

Two papers supply method. EPSVec shows in text generation that a domain can be summarized from 10–50 examples as a single activation-space vector, after which synthesis volume costs nothing extra. That is the amortization idea we transfer to vision, with visual feature statistics in place of LLM activations. IntraStyler extracts exemplar style codes and interpolates between them on the sphere to create unseen intermediate styles. It then feeds them to a full pixel generator, whereas we interpolate statistics and never render.

The contrastive component builds on class-conditional cross-domain alignment. Domain Contrast pulls same-class features across domains together and pushes different-class features apart. RPN Prototype Alignment compresses the classes into learnable prototypes to avoid instance-pair costs. Both depend on target pseudo-labels, whose errors are a known failure mode. Our restyled features inherit the source labels exactly, so the contrastive loss is fully supervised. ZoDi's instance-level cosine loss has no negatives, which is the specific difference our ablations test.

Research Program

The method has three stages.

Building the style bank. For each target condition, we sample $K$ images from an off-the-shelf text-to-image diffusion model with a condition prompt ("a street scene at night, dashcam"). No source image, layout, or mask is involved. We pass the images through the frozen early layers of the segmentation backbone, which is the same CLIP-initialized DeepLabv3+ ResNet-50 used by PØDA. We record channel means and standard deviations per spatial patch rather than per image. We then fit a low-rank Gaussian over these patch statistics to form a style bank. The bank also includes spherical interpolations between exemplar statistics, as in IntraStyler, to fill in unseen intensities.

Restyling source features. During training, each source feature map $f$ is restyled patch-wise. For each patch $p$, the feature channels are normalized and then rescaled with statistics sampled from the bank:

$$\tilde f_p = \sigma^{*}_p \frac{f_p - \mu(f_p)}{\sigma(f_p)} + \mu^{*}_p,$$

where $\mu(f_p)$ and $\sigma(f_p)$ are the channel mean and standard deviation of patch $p$ of the source feature map, and $(\mu^{*}_p,\sigma^{*}_p)$ are drawn from the bank. A smooth random mixing mask makes the intensity vary across the image, mimicking non-uniform fog or lighting.

Training objective. The segmentation head is trained with cross-entropy on both $f$ and $\tilde f$. We add a prototype contrastive loss that keeps restyling from moving features across class boundaries:

$$\mathcal{L}_{\text{proto}} = -\sum_{i}\log\frac{\exp(z_i^\top c_{y_i}/\tau)}{\sum_{k}\exp(z_i^\top c_k/\tau)}.$$

Here $z_i$ is a normalized projected feature at pixel $i$, taken from either $f$ or $\tilde f$. $y_i$ is its true source label, $c_k$ is a learnable prototype for class $k$, and $\tau$ is a temperature.

Modality-gap diagnostic. Alongside the method, we measure where each target lands relative to real ACDC feature statistics. The targets are PØDA's text-optimized statistics, a gap-corrected text variant, exemplar statistics, and real ACDC statistics taken from ACDC's training split and used for measurement only. The gap-corrected variant adds the text-space direction from "clear" to "night" to image embeddings instead of targeting the raw text embedding. We use Fréchet distance between statistic distributions and a linear domain classifier for these measurements. This tests whether the modality gap accounts for text's shortfall.

Baselines

The naive reference is source-only training on Cityscapes, evaluated on ACDC. Every method must beat it, and PØDA reports it at 18.31% mIoU on ACDC Night.

The cheap competitor is PØDA, rerun with its released recipe. On ACDC Night it improves mIoU from 18.31% to 25.03% (PØDA), and ZoDi reports it at 43.9 mIoU on Snow and 49.0 on Fog.

The strongest existing method is ZoDi, the per-image diffusion method. It reports 56.1 versus PØDA's 49.0 mIoU on Fog and 45.6 versus 43.9 on Snow (ZoDi). SIDA reports 46.48% mIoU on ACDC Snow and 25.08% mean mIoU on fire and sandstorm (SIDA). Because these numbers come from different backbones and protocols, we treat them only as sanity anchors. All methods are rerun under one protocol: the same backbone, the same Cityscapes training set, the same number of training iterations, and the same hyperparameter search budget, tuned on a held-out generated validation set rather than on ACDC labels. SIDA's full GPT-4o pipeline may be infeasible to reproduce exactly. In that case we run its image-to-image stage with an open captioner and report it as a partial reproduction.

Beating the strong baseline means matching ZoDi within about one mIoU on average across the four ACDC conditions at a total generation cost at least ten times lower. Clearly beating PØDA on every condition would count as a positive result even if the method stays below ZoDi. Costs are measured as GPU-seconds for generation plus training.

Four ablations isolate the key ingredients:

  1. Replace generated exemplars with gap-corrected text statistics ($K=0$). This separates what the visual anchor adds beyond fixing the gap.
  2. Use per-image instead of per-patch statistics. This isolates the spatial non-uniformity claim.
  3. Replace the prototype loss with ZoDi's cosine loss, or remove it entirely. This tests whether pushing away negative classes matters.
  4. Replace generated exemplars with $K$ real ACDC training images. This is an oracle that tells whether synthetic exemplar quality or the statistic-transfer mechanism is the bottleneck.

Why not just…?

Why not just generate the whole training set offline with diffusion, as ZoDi and SIDA do? This is the real accuracy threat. It is a one-time cost, and it currently gives the best reported numbers, such as 56.1 Fog mIoU for ZoDi. The case against it is cost and label integrity. At over a minute per image (PØDA), several thousand Cityscapes images per condition amount to tens of GPU-hours. SIDA's pipeline adds GPT-4o calls, and both papers acknowledge that the generator can alter layouts under the masks. The budget curve from $K=0$ up to per-image synthesis settles this directly. If accuracy keeps climbing all the way to per-image synthesis, the offline approach wins and we will say so.

Why not just fix the text target by correcting for the modality gap? Mind the Gap suggests the gap behaves largely as a consistent offset, so a directional text target might recover much of the loss at zero generation cost. This is a serious alternative, so we run it as ablation 1 and include it on the budget curve as the $K=0$ point. Our expectation, based on SIDA's evidence that text misses non-uniform weather effects, is that it closes part of the PØDA-to-ZoDi gap but not all of it. If it matches the exemplar method, the paper's conclusion becomes "correct the gap, skip generation," and that is still a useful result.

Why not just retrieve real target-condition images from a web pool, as MUDG does? MUDG retrieves across the modality gap from LAION-2B in seconds. Night and snow street scenes are plentiful on the web, so for common conditions retrieval may equal generation as an exemplar source. We will test retrieved exemplars in place of generated ones. We expect retrieval to lose on rare conditions such as fire and sandstorm, which MUDG itself notes as a failure mode for out-of-vocabulary concepts. Retrieval and generation are therefore interchangeable exemplar sources within our method, not competitors to it.

Why not just self-train on the $K$ generated images as unlabeled target data? Generated images have no masks, so they can only be used through pseudo-labels. Pseudo-label errors are exactly the failure mode that cross-domain contrastive methods struggle with. With only tens of images, self-training would also overfit their content. We run this as a baseline at matched $K$. Our method transfers statistics onto thousands of exactly labeled source layouts and should win clearly. If it does not, the value lies in the exemplar images themselves rather than in the transfer mechanism.

Proposed Experiments

The main experiment trains on Cityscapes and evaluates mIoU on the ACDC Night, Snow, Rain, and Fog validation sets (ACDC), with fire and sandstorm prompts as a secondary stress test following SIDA. The central figure plots mIoU against total generation GPU-seconds for $K\in\{0,4,16,64,256\}$, for per-image diffusion, and for PØDA. Each point is averaged over three seeds and three independently generated exemplar sets, so that sampling variance in the few exemplars is visible.

Three outcomes are decisive for the core claim. The first is whether the curve saturates by $K$ in the tens. The second is whether the generated-exemplar curve tracks the real-ACDC oracle at matched $K$, which would show that synthetic exemplars are adequate anchors. The third is whether patch statistics beat image-level statistics most on Fog and Night, where effects are least uniform.

The gap diagnostic is judged by its own test. If gap-corrected text statistics land measurably closer to real ACDC statistics than PØDA's raw text statistics, and that distance ordering predicts the mIoU ordering across all targets, then the modality gap explanation holds. If not, text's weakness is not geometric but simply informational, and we will report it that way.

Finally, the prototype-loss ablation is checked per class. We expect restyling without negatives to cause confusions between visually similar classes, such as road versus sidewalk at night. Class-level IoU should show whether the contrastive term is what prevents them.