Detectors trained on frames from a video game often fail on real dashcam footage, because rendered images have a recognizable look: clean edges, flat lighting, uniform textures. A cheap fix is now available. Take each synthetic frame, add some noise, and let a pretrained text-to-image diffusion model (a generator that turns noise into an image by removing noise step by step) redraw it as a photograph. The model is told what to aim for ("a dashcam photo") and what to avoid ("video game graphics, CGI render"). Recent work on negative prompts complicates this. A negative prompt applied too early in the denoising process can produce the very concept it was meant to remove, an effect called reverse activation (Understanding the Impact of Negative Prompts). In a sim-to-real pipeline, two things could go wrong. The model could make images look more game-like, not less. It could also bend scene geometry so that the free pixel-accurate labels from the game engine no longer match the image.
This note proposes measuring, and then scheduling, when negative guidance switches on during image-anchored synthetic-to-real translation, with downstream detection accuracy as the arbiter. It also adds a constraint that nobody in this space has checked. Web-trained diffusion models have plausibly seen the real benchmark images, such as KITTI and Cityscapes street scenes, that sim-to-real methods are scored on. A "translation" that looks like it closes the domain gap may partly be replaying memorized test-domain imagery. The work therefore uses benchmark contamination as an evaluation lens, not as a third method.
Core Claim
The timing rules found for generation from pure noise do not transfer unchanged to translation that starts from a partially noised synthetic image. In the from-scratch setting, Ban et al. report that noun-level negative prompts work best from around step 5 and attribute-level ones plateau around step 10. Earlier application distorts the forming layout and triggers reverse activation. In image-anchored translation, the layout is already present in the starting latent, so the safe window should depend on the starting noise level. "Video game look" behaves like an attribute, not an object, so its window should resemble the later, attribute-like one. We hypothesize the following. A negative prompt delayed by a fixed fraction of the remaining denoising steps will beat two alternatives: always-on negative guidance, and the step-0 synchronized suppression of Enhancing Physical Plausibility in Video Generation by Reasoning the Implausibility. It will win on two counts: residual "synthetic-ness" and label fidelity. These gains should then appear as higher real-world detection accuracy.
The guidance rule we schedule is the standard negative-prompt form of classifier-free guidance:
$$\hat\epsilon_t = \epsilon_\theta(x_t, c_{\text{neg}}) + w\,\big(\epsilon_\theta(x_t, c_{\text{pos}}) - \epsilon_\theta(x_t, c_{\text{neg}})\big).$$
Here $x_t$ is the noisy latent at step $t$, $\epsilon_\theta$ is the model's noise prediction, $c_{\text{pos}}$ and $c_{\text{neg}}$ are the positive and negative text embeddings, and $w$ is the guidance weight. Outside the scheduled window, $c_{\text{neg}}$ is replaced by the empty-prompt embedding. The object of study is the window's start, expressed as a fraction $\rho$ of the steps remaining after the starting noise strength $s$.
This setting also gives a task-grounded way to settle a dispute in the literature. Ban et al. argue that negative guidance must wait for structure to form. Hao et al. argue that the core problem is that negative guidance arrives too late, and they force it on from step 0 with directional normalization. Both judge results by perceptual or model-judged scores. A detector's mean average precision on real images, with labels inherited from the simulator, is a harder referee, because it penalizes both residual game-look and geometric drift.
Research Program
The work has three parts.
Part 1: Map reverse activation for style concepts in image-anchored translation. We start from Sim 10k frames produced by the GTA V rendering hooks of Driving in the Matrix. We sweep starting strength $s$, guidance weight $w$, and window start $\rho$. For each output we record three measurements. The first is a residual synthetic score: the probability, under a held-out sim-versus-real classifier, that the image is synthetic. We operationalize reverse activation as the fraction of images whose synthetic score rises relative to positive-prompt-only translation at the same $s$. The second is label fidelity: mask overlap between a frozen off-the-shelf segmenter's vehicle masks on the translated image and the simulator's stencil masks. The third is cross-attention maps for the negative tokens, following Ban et al.'s analysis, to check whether early guidance acts through the "inducing" and "momentum" mechanisms they describe.
Part 2: Turn the map into a schedule. The schedule is a single rule $\rho^*(s)$ fit on Sim 10k and frozen. We then test it unchanged on a second, structurally different domain: procedurally generated retail shelves from 3D-IRAIS, translated toward Real-IRAIS appearance (Amodal Instance Segmentation with IRAIS Dataset for Sim-to-Real Transfer). The retail domain matters for two reasons. Occlusion-heavy scenes make geometric drift expensive, since amodal masks of hidden object parts cannot be re-derived from the image. Real-IRAIS is also recent, which leads into Part 3.
Part 3: Run the contamination controls. These test whether translation gains reflect domain transfer rather than recall of benchmark imagery. For every translated frame we retrieve its nearest neighbors, by perceptual embedding, among KITTI and Cityscapes training and test frames. We then report detection gains separately for frames with near-duplicate matches and for frames without them. We also compare generic positive prompts ("a dashcam photo") with dataset-evoking ones ("a KITTI street scene, Karlsruhe"). If the evocative prompt helps on KITTI specifically, that is a warning sign. Finally, we collect a small held-out real driving set recorded after the diffusion model's training cutoff, annotated for vehicles. It serves as a leakage-free test that no model could have seen. This follows the logic of Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?, which built a post-cutoff test set because performance on older static benchmarks overstated reasoning. We carry that logic from text to images. DyVal motivates the same caution through a different route: it showed static benchmark leaders collapsing on freshly generated problems. We do not attempt DyVal-style procedural generation of real images, which is not possible. A post-cutoff capture is the honest visual analogue.
Baselines
The do-nothing reference trains Faster R-CNN, the detector family used by Johnson-Roberson et al., on raw Sim 10k and evaluates on KITTI Easy, Moderate and Hard. The paper reports that Sim 10k underperformed its larger sets, but no reported result found on Lacuna for Sim 10k's exact KITTI mAP, so we rerun it ourselves.
The strongest existing data-side reference is the same detector trained on raw Sim 200k. Driving in the Matrix reports 0.6803 mAP on KITTI Easy and 0.4207 on KITTI Hard, against 0.6247 and 0.3566 for training on real Cityscapes. Our headline target is this: Sim 10k translated with the timed schedule should match or exceed Sim 200k raw on KITTI Hard and on the post-cutoff set, using one-twentieth the synthetic frames. To keep the comparison fair, we retrain both under identical detector settings and iteration budgets rather than relying only on the 2016 numbers.
The strongest existing translation method is the pixel-level adversarial approach of PixelDA. It maps synthetic images to the target look with a content-similarity loss that protects foreground geometry, and reports 98.2% on MNIST to MNIST-M and a 23.5° pose error on LineMod. It is the most direct competitor because it explicitly guards labels while changing appearance. We train it on the same unlabeled real images our diffusion pipeline is allowed to see, with a matched GPU-hour budget. There is no reported result found on Lacuna for PixelDA on Sim 10k to KITTI.
The guidance-design competitor is calibrated classifier guidance from Elucidating The Design Space of Classifier-Guided Diffusion Generation. It steers sampling with an off-the-shelf classifier applied to the predicted clean image, with Softplus-smoothed gradients and calibration. That paper reports an ImageNet 128×128 FID of 2.19, against 2.43 for classifier-free guidance. Here the classifier is a calibrated sim-versus-real network that pushes away from "synthetic," run through the EluCD codebase. This tests whether a gradient signal sidesteps the timing problem entirely.
Two schedule competitors are run within our pipeline at identical $s$, $w$ and sampler. One is always-on negative prompting. The other is Hao et al.'s Synchronized Decoupled Guidance, which forces suppression from step 0.
Two ablations isolate the key ingredient. Positive-prompt-only translation tells us whether the negative prompt contributes at all. Ban et al.'s fixed from-scratch step offsets, transplanted naively, test whether the noise-dependent rule $\rho^*(s)$ matters.
Why not just…?
Why not just generate more synthetic data or randomize the simulator harder? Scale is a real threat. Johnson-Roberson et al. show that diversity thresholds alone take Sim 200k past real Cityscapes training. DEAL reports that domain-randomized quadruped controllers "frequently failed or drifted significantly" on hardware, but that evidence is about physical dynamics, not appearance. We address this directly by comparing translated Sim 10k against raw Sim 200k under equal detector training. If raw scale wins on the post-cutoff set, the proposal's practical value reduces to data efficiency, and we will say so.
Why not just skip the negative prompt and lower the denoising strength? This may be the most dangerous alternative. A positive prompt at modest strength may already remove most of the game look while preserving labels, making negative timing irrelevant. The positive-only ablation, swept over the same $s$ grid, settles it. The proposal claims value only if timed negatives move the Pareto front of residual synthetic score against label fidelity outward at some strength, and if that shift raises mAP.
Why not just use a GAN translator like PixelDA? PixelDA protects foreground geometry well but is described as handling low-level appearance rather than structural variation. It also needs per-domain adversarial training. The matched-budget comparison will show whether a pretrained diffusion model's broader knowledge of real-world appearance is worth its risks. The retail transfer test asks whether either method holds up when occlusion structure matters.
Why not just fine-tune the detector on a little real data? Real annotation is the bottleneck these pipelines exist to avoid, as with amodal masks in retail. Still, we include a variant with a few hundred labeled real frames, and report whether translation adds anything on top of it.
Proposed Experiments and Decisive Checks
The main experiment trains detectors on each translated version of Sim 10k. We report mAP on KITTI (Easy, Moderate, Hard), on Cityscapes vehicles, and on the post-cutoff set. For retail, we train EUREKA-style amodal segmenters on translated 3D-IRAIS and evaluate amodal AP on Real-IRAIS. The reference point there is the reported 37.10% from synthetic-only training, against prior baselines of 14% to 19%.
Four outcomes would be decisive. First, reverse activation exists for style concepts: early or always-on negative guidance raises the residual synthetic score above positive-only translation in a measurable fraction of images, with cross-attention evidence matching Ban et al.'s mechanism. Second, the noise-dependent schedule beats both always-on and step-0 synchronized guidance on KITTI Hard and on the post-cutoff set. Third, the frozen schedule transfers to the retail domain without retuning. Fourth, and most important for credibility, the gains survive contamination controls. They must be similar on frames without near-duplicate KITTI matches and hold on post-cutoff data, while dataset-evoking prompts give no KITTI-specific advantage. If gains concentrate on near-duplicate frames or vanish on fresh data, the finding is the reverse but equally publishable: diffusion-based sim-to-real results on public benchmarks are inflated by leakage from pretraining. That would argue for post-cutoff test sets as standard practice.
References
- Understanding the Impact of Negative Prompts: When and How Do They Take Effect?
- Enhancing Physical Plausibility in Video Generation by Reasoning the Implausibility
- Driving in the Matrix: Can virtual worlds replace human-generated annotations for real world tasks?
- Amodal Instance Segmentation with IRAIS Dataset for Sim-to-Real Transfer
- Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks
- Elucidating the design space of classifier-guided diffusion generation
- DEAL: Diffusion Evolution Adversarial Learning for Sim-to-Real Transfer