A cheap way to get labeled training data for self-driving perception is to render scenes in a video game and then "repaint" them so they look like real camera footage. The pixel labels come for free from the game engine. Current repainting tools run inside the compressed internal representation of Stable Diffusion, called a latent space. That representation is produced by an autoencoder that downsamples the image eightfold. The compression step blurs away thin poles, distant pedestrians, traffic signs, and sharp object boundaries. Those are exactly the pixels whose labels matter most and are hardest to get. Blurring them means the repainted image no longer lines up with the game's labels. We propose to replace the latent-space repainter with a one-step model that works directly on pixels. It is trained with a loss that says explicitly which spatial frequencies must come from the rendered source frame (geometry and edges, which carry the labels) and which should be pushed toward real-camera statistics (surface texture, sensor noise, lighting).
The scientific question underneath is a real tension in the literature. In generative vision, the autoencoder bottleneck is treated as purely harmful because it destroys fine detail. In robot sim-to-real transfer, compressing what the model sees is treated as helpful because it strips away simulator-specific cues. We want to find out which view holds for synthetic-to-real segmentation, and for which objects. The engineering payoff, if the band-selective translator wins, is repainted data that stays aligned with its labels, produced in one forward pass instead of a multi-step diffusion inversion.
Core Claim
The autoencoder in latent-diffusion style transfer acts as an uncontrolled filter. It removes all high-frequency content at once, including both simulator artifacts and label-bearing structure. We claim a translator that works in pixel space and enforces a per-frequency contract will transfer better to real driving scenes, with the largest gains on small and thin classes and at object boundaries. The contract keeps label-aligned structure pinned to the rendered frame and matches texture and noise to real images. We also claim the translator can be distilled into a single step at a small fraction of the cost of multi-step translation. Nothing in the file does both halves. The closest method translates in latent space with many steps. The VAE-free and one-step methods have only been used for super-resolution or low-resolution classification augmentation.
Grounding in Prior Work
Work that motivates the proposal. Several papers show the bottleneck erasing detail:
- The frequency-spectrum study Harnessing Frequency Spectrum Insights for Image Copyright Protection Against Diffusion Models finds the VAE attenuates high-frequency patterns.
- DiffUTE and TextInVision show that the same downsampling garbles fine strokes.
- RORem shows that an encode–decode round trip shifts unedited background pixels.
For sim-to-real data, a spatial shift means a label error.
The closest method. Style Transfer with Diffusion Models for Synthetic-to-Real Domain Adaptation (CACTI/CACTIF) repaints GTA5 frames toward Cityscapes and ACDC. It uses DDPM inversion in Stable Diffusion's latent space, normalizes style statistics separately for each semantic class, and filters cross-attention to avoid hallucinated structure. It reports more than a 4% gain on Cityscapes over raw synthetic data and 58.24% mIoU on ACDC fog with HRDA. Its best nighttime result is only 26.07% mIoU. It operates entirely inside the lossy latent space, uses no training loss (only inference-time manipulation), and needs many sampling steps per image. We keep its goal and its insight that style should be matched per class. We change where the translation happens and how detail is assigned.
Work that supplies the method. Two restoration papers provide the components:
- Eliminating VAE for Fast and High-Resolution Generative Detail Restoration (GenDR-Pix) removes the autoencoder. It uses lossless pixel-unshuffle, which rearranges pixel blocks into channels, and pixel-shuffle to undo it. It suppresses the resulting checkerboard artifacts with a Masked Fourier Space loss.
- GenDR shows that expanding the latent to 16 channels matters more for detail than enlarging the denoising network. It also gives a one-step distillation recipe that combines pixel, adversarial, and DINOv2 feature-alignment losses.
We reuse the pixel-unshuffle backbone, the Fourier artifact penalty, and the distillation recipe. Neither paper studies domain shift or downstream segmentation.
Work that constrains the design. Three papers limit what the loss can look like:
- On The Classification-Distortion-Perception Tradeoff proves that realism and task utility trade off. Its information-loss principle says processing can only keep or destroy the semantic information already present in the input. For translation with known source labels, the translator cannot add label information, only lose alignment. So the loss must change appearance while protecting structure, and a purely adversarial realism objective is ruled out.
- SINCO shows that adding a frozen segmenter's Dice loss to a reconstruction objective preserves boundaries under heavy compression. We deliberately do not copy it directly. A segmenter trained on synthetic data would reward preserving the very simulator shortcuts we want removed.
- Bridging the Sim-to-Real Gap from the Information Bottleneck Perspective (HIB) finds that compressing privileged simulator information improves physical transfer. It is the strongest evidence for the opposing hypothesis, that the VAE's blurring helps by removing rendering cues. We test that hypothesis directly rather than assume it away.
Related one-step augmentation work. DAR-GDA distills diffusion into a one-step generator (over 100× faster) and reweights synthetic samples with a discriminator's density ratio. It was evaluated only on classification at modest resolution, and it stays in the latent space.
Research Program
We first build a diagnostic. We compute the radially averaged power spectrum of GTA5 frames and of Cityscapes and ACDC frames, and measure where they differ. We do the same for GTA5 frames after a plain Stable Diffusion encode–decode round trip. We also measure label alignment. We run edge detection on each translated image and compute the boundary F-score of those edges against the GTA5 label boundaries, reported separately by object size and class. This shows how much of the "domain gap" the autoencoder closes by accident, and how much label alignment it costs.
We then build the translator. The backbone follows GenDR-Pix: a Stable Diffusion 2.1 UNet operating on pixel-unshuffled input, initialized from pretrained weights. Its teacher is CACTIF, run on a subset of roughly 2,000 GTA5 frames per target condition. Let $x_s$ be a rendered source frame, $G(x_s)$ the one-step output, and $x_t$ the teacher's translation. Let $L_k$ and $H_k$ be low-pass and high-pass filters at cutoff frequency $k$, and let $B$ be a mask that dilates the GTA5 label boundaries by a few pixels. The objective is
$$\mathcal{L} = \| L_k (G(x_s) - x_t) \|_1 + \alpha \, \| B \odot (\nabla H_k G(x_s) - \nabla H_k x_s) \|_1 + \beta \, \mathcal{L}_{adv}(G(x_s); \text{real}) + \gamma \, \mathcal{L}_{MFS}(G(x_s)).$$
In words, each term does one job:
- The first term copies the teacher's appearance only at low frequencies. The student therefore inherits the global color, lighting, and weather layout, but not the teacher's blurred detail.
- The second term forces high-frequency gradients near label boundaries to match the rendered frame. This keeps edges exactly where the labels say they are.
- The adversarial term uses a discriminator that sees only high-pass-filtered patches of unlabeled real images. It pushes texture and sensor noise away from boundaries toward real-camera statistics. Real-image noise is the main thing a clean renderer lacks.
- The Masked Fourier Space penalty from GenDR-Pix suppresses pixel-unshuffle checkerboards.
We follow CACTI and compute the low-frequency match per class. The cutoff $k$ is chosen from the phase-one diagnostic, at the band where GTA5 and real spectra begin to diverge. We also sweep it.
The central scientific experiment is a bottleneck sweep. We train the identical translator and loss with three representations: a 4-channel VAE latent, GenDR's 16-channel SD2.1-VAE16 latent, and lossless pixel-unshuffle. If the HIB-style view is right, the 4-channel version should transfer as well or better on large, texture-driven classes. If the detail-erasure view is right, pixel space should win on small and thin classes. The answer can also differ by class size, which is itself a result: the bottleneck's effect would depend on object scale.
Baselines
The naive reference point is segmentation trained on raw GTA5 frames with standard photometric and geometric augmentation. We use the same Segformer and HRDA architectures, schedules, and Cityscapes or ACDC unlabeled images as every other arm. Playing for Data: Ground Truth from Computer Games established that this data alone raises CamVid mIoU by 3.9 points when mixed with real data. Any translator must beat raw frames under identical training to justify itself.
The strongest existing method is CACTIF from Style Transfer with Diffusion Models for Synthetic-to-Real Domain Adaptation. Its reported numbers are over 4% Cityscapes improvement over raw synthetic data with Segformer, 58.24% mIoU on ACDC fog with HRDA, and 26.07% mIoU as the best on ACDC night. We run it with its released settings on the full GTA5 set for each target condition, and train segmenters identically on its outputs. We report both our reproduced numbers and theirs. Beating it means higher mean mIoU on Cityscapes and on at least three of the four ACDC conditions, averaged over three segmenter seeds, with a margin larger than the seed spread. It also means a clear gain on small classes (pole, traffic light, traffic sign, person, rider, bicycle) and on boundary F-score. A mean-mIoU gain coming only from large classes like road and sky would not support our claim.
Two stronger variants of CACTIF test whether a simpler detail fix suffices. The first swaps in the 16-channel SD2.1-VAE16 from GenDR. The second adds encoder-to-decoder skip connections in the style of FreeAdapt, which let source detail bypass the latent.
The ablations isolate the key ingredient:
- No boundary term ($\alpha = 0$): tests whether the edge-pinning term is what preserves label alignment.
- Full-band regression to the teacher (the low-pass filter removed from the first term): tests whether copying the teacher's blurred detail is what we are avoiding.
- Full-image adversarial discriminator (not restricted to high-pass patches away from boundaries): tests whether confining realism pressure to texture matters.
- SINCO-style frozen-segmenter Dice term in place of the edge term: tests the shortcut-preservation worry.
All arms get the same number of translated images and the same segmenter training budget. Every translator's hyperparameters are tuned with the same number of trials on a held-out split of Cityscapes train.
Why not just…?
Why not just use a bigger latent, like the 16-channel VAE, inside CACTIF? This is the most serious threat. GenDR's result that latent channel count governs detail fidelity suggests it could recover most of the loss. The CACTIF-plus-VAE16 baseline and the 16-channel arm of the bottleneck sweep settle it directly. If VAE16 matches pixel space on small-class IoU and boundary F-score, we will report that the representation matters less than the band-selective loss. The contribution then narrows to the loss and the one-step cost, and we will present it that way.
Why not just translate with a training-free frequency swap, or paste the source's high frequencies back onto CACTIF's output? Both are nearly free and would perfectly preserve edge alignment. The first replaces the low-frequency Fourier amplitude of GTA5 frames with that of real frames. The second recombines CACTIF's low frequencies with the source frame's high frequencies. However, pasting back the rendered high frequencies also pastes back the renderer's clean, noise-free texture. That texture is the shortcut family described in Jet substructure classification in high-energy physics with deep neural networks and in the synthetic-data literature. It is also exactly what our high-pass discriminator is meant to replace. We include the paste-back as a baseline. If it matches our translator, the learned texture term is unnecessary, and we would say so.
Why not just run CACTIF offline once, since data generation is a one-time cost? For a single target domain and 25k frames, this argument largely holds. We will not claim speed as the main win in that setting. Cost becomes decisive when a practitioner needs many target conditions, several style references, or fresh translations each epoch. Diffusion Curriculum shows that varied, adaptively chosen synthetic samples improve training, but it pays multi-step sampling costs to produce them. We report total GPU-hours including teacher runs and distillation. We also run an experiment that resamples translations with a new real style reference each epoch. That experiment is only affordable with a one-step model, and it checks whether the extra variety improves ACDC transfer.
Why not just reweight synthetic samples by a density ratio, as DAR-GDA does? Reweighting corrects a mismatch in the distribution of whole images. It does not repair per-pixel misalignment between an image and its label map. DAR-GDA also leaves open whether density ratios can be estimated reliably for high-resolution scenes. We include it as an optional add-on to the best translator rather than a competitor.
Proposed Experiments
We translate the full GTA5 set toward Cityscapes and toward each ACDC condition (fog, night, rain, snow). We train Segformer and HRDA on each translated set and evaluate on Cityscapes val and ACDC val. We report:
- mIoU overall and for the small-class group;
- boundary F-score of the segmenter's predictions;
- the translated images' own label alignment, from the phase-one boundary F-score;
- FID against real target images;
- GPU-seconds per 1024×2048 frame, and total GPU-hours including distillation.
Three checks are decisive:
- Bottleneck sweep: if the pixel-space arm does not beat the 4-channel arm on small-class IoU, the detail-erasure hypothesis is wrong for this task.
- Frequency-alignment check: the translator's output spectrum should match real spectra above the cutoff away from boundaries, while tracking the source spectrum at boundaries. We will show this with spectra computed separately inside and outside the boundary mask.
- Night stress test: because 26.07% mIoU is the reported ceiling on ACDC night, we test whether pinning geometry lets us use a much stronger appearance change there without the structural hallucination that limits CACTIF.
A negative result on night would still show where band-selective translation stops helping. A positive one would locate part of that failure in the latent bottleneck rather than in the difficulty of the domain itself.
References
- Harnessing Frequency Spectrum Insights for Image Copyright Protection Against Diffusion Models
- DiffUTE: Universal Text Editing Diffusion Model
- TextInVision: Text and Prompt Complexity Driven Visual Text Generation Benchmark
- RORem: Training a Robust Object Remover with Human-in-the-Loop
- Style transfer with diffusion models for synthetic-to-real domain adaptation
- Eliminating VAE for Fast and High-Resolution Generative Detail Restoration
- GenDR: Lighten Generative Detail Restoration
- On The Classification-Distortion-Perception Tradeoff
- SINCO: A Novel structural regularizer for image compression using implicit neural representations
- Bridging the Sim-to-Real Gap from the Information Bottleneck Perspective
- Generative Data Augmentation via Diffusion Distillation, Adversarial Alignment, and Importance Reweighting
- Playing for Data: Ground Truth from Computer Games
- FreeAdapt: Unleashing Diffusion Priors for Ultra-High-Definition Image Restoration
- Jet substructure classification in high-energy physics with deep neural networks
- Diffusion Curriculum: Synthetic-to-Real Generative Curriculum Learning via Image-Guided Diffusion