Training a vision model on synthetic images from a diffusion model (a generator that turns random noise into images by gradually denoising it) is now a standard way to make detectors and segmenters hold up on datasets they never saw: night instead of day, fog instead of clear skies, real streets instead of video-game renders. It is also expensive. Every synthetic image costs dozens of denoising passes through a billion-parameter network. Much of that budget may be wasted, because a large share of a typical "domain shift" is a change in low-level color and texture statistics. Augmentations that cost almost nothing already simulate that kind of change. We propose to measure, before generating at scale, how much of each candidate shift a model trained with cheap augmentation already handles. We would then spend the diffusion budget only on the shifts where it still fails.
The core claim has two parts. First, cross-dataset gains from diffusion-generated data, plotted against GPU-hours, should rise sharply on a few target conditions and stay flat on the rest. Second, a short pilot can tell these apart in advance. The pilot measures how badly a cheaply augmented model does on a small batch of generated images, after subtracting the loss it suffers on generated images that contain no shift at all. If the claim holds, practitioners get most of the benefit of full-scale synthesis for a fraction of the compute. If it fails, because the pilot signal does not predict gains or because cheap augmentation matches synthesis once compute is equal, that is also a useful answer to an open dispute in the literature.
Why this question, and why now
The motivating tension is stated sharply in the material. On one side, explicit synthesis pipelines report large cross-dataset gains at large cost. Let Synthetic Data Shine (DRSF) generates several "pseudo-target" weather domains from daytime driving scenes with InstanceDiffusion. It then suppresses noisy synthetic feature channels and reaches 34.7% mean performance across targets on the Diverse Weather Dataset. S3OD generates 139,000 high-resolution images to cut cross-dataset error by 20–50%. LangDAug bridges medical imaging sites with energy-based sampling whose training cost grows quadratically in the number of source domains.
On the other side, MixStyle simulates new domains by mixing the per-channel mean and standard deviation of early-layer features between images, and it costs almost nothing. An Analysis of Model Robustness across Concurrent Distribution Shifts evaluates 26 algorithms over 168 shift pairs and finds that RandAug acts as a "no-regret" baseline that often beats complex debiasing methods. None of these papers compares the two families at equal compute. DRSF itself does not report how its gains scale with the number of generated domains or with generation GPU-hours.
Two further findings constrain the design. DRSF documents a "synthetic data paradox": generated images carry non-physical artifacts that form their own clusters in feature space. Any signal that asks "is this generated image novel to my model?" will therefore partly answer "does this image look generated?" The concurrent-shift analysis shows that a single hard component dominates error when shifts combine. This suggests that compound conditions such as night plus rain are where a budget should concentrate, which is a prediction our allocator can be checked against.
Core Claim
Let $L_c(\cdot)$ denote the task loss of a detector trained on source data with cheap augmentation (RandAug plus MixStyle). The loss is computed on generated images whose box labels are inherited from the source layout, which InstanceDiffusion's layout conditioning preserves. For a candidate condition $d$, such as "foggy dusk," we generate a pilot set $P_d$ of a few hundred images. We also generate a control set $P_0$: the same source scenes re-rendered with a prompt describing the source condition itself, so the control carries diffusion artifacts but no domain shift. The residual for condition $d$ is
$$R(d) = L_c(P_d) - L_c(P_0),$$
the extra loss on condition $d$ that cheap augmentation does not absorb, with the artifact penalty subtracted. Given a total generation budget of $B$ images, the full-scale budget is allocated in proportion to $\max(R(d), 0)$. The same computation can run at finer granularity, over clusters of source scenes crossed with conditions.
The claim is that $R(d)$ ranks conditions by the real-target gain that synthesis on $d$ would deliver. It follows that residual-allocated synthesis at a fraction of DRSF's generation budget matches full DRSF, and beats uniform allocation at every equal budget.
This differs from the closest existing work in a specific way. S3OD also steers generation, using the failure cases of its own model. That signal is measured against a model that never saw cheap augmentation, and it does not separate failures caused by real shift from failures caused by generation artifacts. Our residual is conditioned on what cheap augmentation already covers and corrected for artifacts. It is also evaluated as a compute-allocation rule rather than as a quality improvement.
Research Program
The first component is a compute-accounted reproduction. We will reproduce DRSF on the Diverse Weather Dataset (single-source detection, trained on Daytime-Clear and tested on five unseen weather conditions). We will also reproduce it on GTA5 to Cityscapes, BDD100K and Mapillary (synthetic-to-real segmentation). We will log GPU-hours for diffusion sampling, any filtering, and training. Every result in the study is reported as a point on a curve of target performance against total GPU-hours, never as a single number.
The second component is the decomposition. Keeping DRSF's fusion and channel-suppression modules fixed, we vary the synthetic budget from 0% to 100% of the reproduced generation volume, with and without cheap augmentation in the base training. This tells us directly how much of DRSF's gain over plain training cheap augmentation already reaches, and where the curves cross.
The third component is the allocator. We will build a pool of about a dozen prompted conditions, which includes the five DWD-like conditions, compound conditions, and conditions with no matching test set. We compute $R(d)$ from pilots and allocate the budget. The pilot cost (generation plus one forward pass of the cheap model) is charged against the same budget.
The fourth component is a backbone arm. We repeat the decomposition with a vision foundation model backbone. Cekmeceli et al. report that such models stay stable under scanner changes where task-specific networks degrade, so the residual may shrink toward zero. If that happens, the honest conclusion is that strong pre-training removes most of the case for synthesis.
Baselines
The naive reference point is a Faster R-CNN trained on Daytime-Clear alone, with the same backbone and schedule as every other arm. On segmentation, the naive reference is the matching segmentation network trained on GTA5 alone.
The cheap competitor is the same model trained with RandAug and MixStyle, inserted in early layers as MixStyle prescribes. It is tuned over the same number of hyperparameter trials as the synthetic arms. On cheap augmentation for DWD we found no reported result on Lacuna for MixStyle or RandAug, so these numbers will come from our runs.
The strongest existing method is DRSF. It reports 34.7% mean performance across targets on the Diverse Weather Dataset, against a prior best of 32.6%, and 48.13% average mIoU across the three real segmentation targets (DRSF). No generation compute is reported for it on Lacuna. We will run DRSF with its full synthetic set and report our reproduced number next to the published one. Beating it means matching its target performance within seed noise at no more than half its total GPU-hours, or exceeding it at equal GPU-hours.
The key ablation is uniform allocation: DRSF's pipeline given the same budget as our allocator, spread evenly over the same condition pool. A second ablation is failure-steered allocation without the artifact control. It drops the $L_c(P_0)$ term and uses a plain-trained rather than cheaply augmented model, which approximates S3OD-style steering. A third ablation keeps the artifact control but scores conditions with the plain-trained model. Together these isolate the two ingredients: conditioning on cheap augmentation, and subtracting artifacts.
All arms share data, detector, training steps, and tuning budget. Every arm uses DRSF's training-time fusion modules, which DRSF reports add 3.8% parameters and are switched off at inference, so differences come only from which images were generated.
Why not just…?
Why not just use RandAug and MixStyle and skip synthesis? This is the most serious threat. The concurrent-shift study's no-regret finding supports it. MixStyle beat Mixup by 5.2% on PACS but makes no claim about nighttime detection. The decomposition experiment settles it directly. If the cheaply augmented model sits on or above the synthesis curve at equal GPU-hours on both benchmarks, we will report that the measured value of diffusion synthesis for weather shift is small, and the allocator becomes moot. We expect it to fail on illumination changes such as night, where MixStyle's restriction to first- and second-order statistics in early layers cannot add headlights, reflections or darkness-induced occlusion. That is a prediction, not a result.
Why not just generate fewer images uniformly? If diminishing returns are smooth and similar across conditions, uniform subsampling captures most of the saving with no pilot machinery. The uniform-allocation ablation tests exactly this. Our idea is only justified if the residual-allocated curve lies measurably above the uniform curve at matched budgets, and if $R(d)$ rank-correlates with per-condition gains.
Why not just steer generation by the model's failures, as S3OD does? Failure steering is close to what we propose. The difference only matters if artifacts and already-covered shifts actually distort failure signals. The two steering ablations show whether subtracting the re-render control and conditioning on the cheap model change the allocation and the final score. If they do not, the simpler S3OD-style rule wins and we will say so.
Why not just use a foundation model backbone? It may erase the residual. The backbone arm measures this, and a near-zero residual under that backbone would be reported as a main finding.
Decisive Checks
The first check is predictive. For each condition in the pool, we train with synthesis on that condition alone and measure real-target gain where a matching test condition exists. Rank correlation between $R(d)$ and this gain must be clearly positive. It must also be stronger than for the version without artifact control, or the core mechanism fails.
The second check is dominance. Residual allocation must lie above uniform allocation at every one of at least four budget levels, across three seeds, on both DWD and GTA5-to-real.
The third check is compositional. The allocator should place disproportionate budget on Night-Rainy-type compound conditions, consistent with the concurrent-shift finding that the hard component dominates. Gains there should account for a large share of the improvement in mean performance.
The fourth check is the backbone arm, which tells us whether any of this survives strong pre-training. A positive outcome on all four would give practitioners a cheap test to run before committing GPU-weeks to generation. A negative outcome on the first two would still settle, at equal compute, a question the literature currently answers only by assertion.
References
- Let Synthetic Data Shine: Domain Reassembly and Soft-Fusion for Single Domain Generalization
- S3OD: Towards Generalizable Salient Object Detection with Synthetic Data
- LangDAug: Langevin Data Augmentation for Multi-Source Domain Generalization in Medical Image Segmentation
- Domain Generalization with MixStyle
- An Analysis of Model Robustness across Concurrent Distribution Shifts
- Do Vision Foundation Models Enhance Domain Generalization in Medical Image Segmentation?