A detector trained only on clear daytime driving footage usually fails at night, in fog, or in rain. A common fix is to use a diffusion model, a generator that turns random noise into images by removing noise step by step, to repaint the training photos in those unseen conditions. This works, but it is expensive. Every synthetic image costs dozens of full denoising passes. Driving images also have to be shrunk to the generator's native size of about 512 pixels, and at that size the small, distant cars and pedestrians that matter most turn into a few blurry pixels. Current pipelines spend the same large budget on every image whether or not that image teaches the detector anything new.
This note proposes a small decision model that, before any pixels are generated, chooses three things for each source image: which weather or lighting condition to paint, how far to push the repainting, and how many denoising steps to spend. It makes these choices at the image's native resolution, using a cheap high-resolution adapter on a frozen generator. The decision model is rewarded only when the generated image is both hard for the detector being trained and still faithful to the original object boxes. That reward is divided by the GPU time the image cost. The claim to test is that most of the generalization gain from synthetic pseudo-domains comes from a small, predictable subset of (image, condition, strength) choices. If so, a learned allocator should match the accuracy of uniform generation at a fraction of the compute, and the native-resolution generation it makes affordable should show up specifically in small-object accuracy.
Core Claim
Two lines of evidence motivate the idea. Let Synthetic Data Shine (DRSF) generates foggy, rainy and night versions of single-source driving images with a layout-preserving diffusion model and reaches 34.7% mean performance across target domains on the Diverse Weather Dataset, against 32.6% for the prior best. The same paper documents a "synthetic data paradox": realistic synthetic images form isolated feature clusters that hurt transfer unless they are suppressed afterward using prediction entropy. DRSF therefore pays full generation cost for every image and then discounts many of them after the fact. AdaGen shows the other half of the picture. A tiny policy network trained with proximal policy optimization over a frozen generator can choose per-sample sampling parameters and cut DiT-XL inference cost threefold. AdaGen, however, optimizes perceptual quality on in-distribution ImageNet and COCO generation. DRSF's paradox says perceptual quality is the wrong target for training data.
The proposal therefore makes AdaGen-style per-sample control answer to a different question: how useful is this image to the learner, per unit of compute? The control decision is moved upstream of generation, so rejected candidates are never rendered. The resolution problem is handled with the mechanism from Make a Cheap Scaling (Self-Cascade). That work adapts a 512-pixel Stable Diffusion to 1024 and 2048 pixels with 0.002M trainable parameters in 10k steps. It does so by noising a low-resolution "pivot" back to an intermediate step $K$ and then letting time-aware feature upsamplers finish at high resolution. We substitute the real source photograph for the generated pivot. In that setting $K$ becomes exactly the knob that trades style change against preservation of the labeled layout. Self-Cascade fixes $K$ by rule. Here it is chosen per image.
Research Program
The generator is Stable Diffusion with the Self-Cascade upsampler adapter. The adapter is trained once on unlabeled source-domain driving images at native resolution (1280×720 for the Diverse Weather Dataset, 2048×1024 for Cityscapes) and then frozen. Generating a pseudo-target image from source image $x$ means four steps: downsample $x$, noise it to step $K$, and denoise it with a condition prompt $c$ for $S$ steps through the cascade, finishing at full resolution. The original boxes are kept as labels.
The allocator is a contextual bandit, not a multi-step reinforcement learning agent. It sees a pooled feature of $x$ from the current detector together with the detector's per-condition running loss, and it outputs a distribution over a discrete grid of $(c, K, S)$. We treat it as a one-shot decision on purpose. The four-model memory cost of on-policy fine-tuning documented by P-TA comes from needing a policy copy, a frozen reference copy, a value network and a reward model. With a one-step decision over a frozen generator, no value network or reference copy is required. The "reward model" is simply the detector we are already training. The decision model is a few-thousand-parameter network. The heavier variant, an AdaGen-style policy that also schedules the denoising trajectory step by step, is evaluated as an alternative rather than assumed.
The reward for a generated image $\tilde{x}$ is
$$r(\tilde{x}) = \frac{\ell_{\text{det}}(\tilde{x}, b)\cdot g(\tilde{x}, x, b)}{\tau(K,S)},$$
where $\ell_{\text{det}}$ is the current detector's loss on $\tilde{x}$ with the source boxes $b$, measuring how hard the image is. The term $g \in [0,1]$ is a validity gate equal to the agreement between high-pass-filtered edge maps of $\tilde{x}$ and $x$ inside the boxes. It reuses the observation behind the Fourier branch of Cauvis that object contours survive style change while color and illumination do not. The denominator $\tau$ is the measured GPU-seconds for that configuration. The gate matters because a pure hardness reward is trivially gamed: destroying an object makes the loss high. This is the downstream-task version of the reward hacking AdaGen reports for FID-driven rewards.
Training alternates between two phases. In the first, the allocator proposes configurations for a batch of source images, only those are rendered, and the generated images enter a replay pool. In the second, the detector trains on source images plus the pool, and the losses it records update the allocator's rewards. The detector is a Faster R-CNN, matching the DRSF setting. On top of it we place DRSF's own feature decoupling and soft-fusion modules, so the comparison isolates which images are generated from how they are consumed. Every run logs total generation GPU-hours, allocator overhead, and the full generated-image histogram over $(c,K,S)$.
Baselines
The naive reference point is the source-only Faster R-CNN trained on Daytime-Clear with standard photometric augmentation. DRSF reports a 4.5× gain over this baseline on Night-Rainy car detection, so it sets the floor. A second naive reference is uniform generation: every source image repainted once per condition at a fixed $K$ and $S$ at native resolution, consumed by the same DRSF modules. This is what the allocator must beat on compute at equal accuracy.
The strongest existing generative method is DRSF itself, rerun from its description on the same data split and the same detector, with its 512-pixel generation. On the Diverse Weather Dataset it reports 34.7% mean performance across targets, and on GTA5-to-{Cityscapes, BDD100K, Mapillary} segmentation it reports 48.13% average mIoU. No reported result was found on Lacuna for DRSF's generation GPU-hours, so we measure it ourselves under identical hardware.
The strongest non-generative method is Cauvis, a frozen DINOv2 detector with causal visual prompts. The file reports only relative gains for it (15.9 to 31.4 mAP over prior methods on Cityscapes SDGOD), and no absolute per-domain mAP was found on Lacuna. We therefore rerun it from its description and also report the allocator's data used with a Cauvis-style detector, to test whether the two gains are complementary.
Three ablations isolate the key ingredient. The first is a static-grid oracle: the single best fixed $(c,K,S)$ triple, selected using target-domain validation. This deliberately cheats in the static baseline's favor and bounds what any fixed schedule could achieve. The second is generate-then-filter: uniform generation at the allocator's budget times the observed rejection ratio, followed by DRSF-style entropy filtering. The third is the allocator with the validity gate removed.
All methods share the source data, the detector, the 50-epoch training schedule and the tuning budget of 20 configurations. Beating the field means matching DRSF-level mean target performance with at most 40% of uniform generation's GPU-hours, or exceeding the static-grid oracle at equal GPU-hours without using any target data.
Why not just…?
Why not just use heavy photometric augmentation? Color jitter and blur do not produce night glare or rain occlusion. Beyond Cropping and Rotation finds classical transforms below generative augmentation. More directly, DRSF's 4.5× Night-Rainy gain is measured against exactly this kind of baseline. We still include it, because it is the cost-zero point on every compute-accuracy curve.
Why not just perturb features adversarially instead of rendering images? Latent perturbation in the style of Volpi et al. is cheap but stays within a small ball around the source distribution. PAPT reports generative domain prompting beating feature-based augmentation, for example by 17.40% on PACS Sketch. We include feature perturbation as a cheap baseline on the same detector. If it closes most of the gap on the Diverse Weather Dataset, the case for spending any generation compute weakens and we will report that.
Why not just generate everything once and filter afterward? This is the most credible threat, since filtering is simple and DRSF already does a soft version of it. The difference is where the compute goes. Filtering pays for every rejected image, and the file notes rejection ratios of 50 to 90% in multi-stage synthetic pipelines. The generate-then-filter ablation at matched GPU-hours settles the question directly. If it ties the allocator, the honest conclusion is that cheap prediction of usefulness before rendering is not possible, and the contribution reduces to native-resolution restyling.
Why not just use a full reinforcement learning policy over the denoising schedule, as AdaGen does? Per-step control could save more compute, but it brings back rollouts, a value head and instability for a decision we suspect is mostly made at the image level. We run the AdaGen-style variant at the same budget. The bandit is preferred unless the step-level policy wins by more than its own training cost.
Why not just restyle at 512 pixels and upsample, or restyle in 512-pixel tiles? Upsampling cannot recover pedestrians that occupied a few pixels after downsampling. Tiling is cheaper than Self-Cascade and keeps the layout, because the real image anchors the structure, so it is a real competitor. Its expected failure is inconsistent weather across tiles, such as fog in one tile and clear sky in the next. The decisive measurement is small-object AP (objects under 32 pixels) under all three generation modes at equal GPU-hours. If tiling matches native-resolution generation, we adopt tiling.
Proposed Experiments
The primary benchmark is the Diverse Weather Dataset: train on Daytime-Clear, test on Night, Foggy, Rainy, Dusk-Rainy and Night-Rainy, and report mean target performance and per-domain mAP. The second benchmark is Cityscapes Day-Clear to its foggy, rainy and night splits (Cityscapes), plus Cityscapes-C corruptions. To check that the allocator is not detection-specific, we run segmentation from GTA5 to Cityscapes, BDD100K and Mapillary, using the DAFormer codebase (lhoyer/DAFormer) as the segmenter. The central figure is target accuracy against total generation GPU-hours at 10%, 25%, 50% and 100% of the uniform pipeline's budget, for every method.
Four checks decide the outcome.
The first check asks whether the allocator's curve dominates uniform generation and generate-then-filter at the 25% and 50% budgets by at least one point of mean target performance. If it does not, the core claim fails.
The second check asks whether the allocator beats the target-tuned static-grid oracle without seeing target data. If not, per-image adaptivity adds nothing beyond picking one good $K$.
The third check examines the learned allocation histogram. We expect it to concentrate low $K$ (light restyling) on images with many small objects and high $K$ on simple scenes. We expect it to concentrate conditions on whichever domains the detector currently fails. Uniform or random-looking choices would mean the reward is noise.
The fourth check asks whether removing the validity gate produces higher training reward but lower target accuracy. That result would confirm that hardness alone is hackable and that the edge-agreement gate is doing real work.
Throughout, we report the allocator's own compute separately, to show that the decision model costs a negligible share of the total, so the reinforcement learning overhead we claim to avoid is measured rather than asserted.
References
- Let Synthetic Data Shine: Domain Reassembly and Soft-Fusion for Single Domain Generalization
- AdaGen: Learning Adaptive Policy for Image Synthesis
- Make a Cheap Scaling: A Self-Cascade Diffusion Model for Higher-Resolution Adaptation
- P-TA: Using Proximal Policy Optimization to Enhance Tabular Data Augmentation via Large Language Models
- Towards Single-Source Domain Generalized Object Detection via Causal Visual Prompts
- Beyond Cropping and Rotation: Automated Evolution of Powerful Task-Specific Augmentations with Generative Models
- Generalizing to Unseen Domains via Adversarial Data Augmentation
- Adversarial Domain Prompt Tuning and Generation for Single Domain Generalization
- The Cityscapes Dataset for Semantic Urban Scene Understanding
- BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning