Simulators produce unlimited labeled video for training robots and driving systems, but the frames look synthetic, and models trained on them lose accuracy on real cameras. A common fix is unpaired image-to-image translation: a model learns to repaint synthetic frames so they look real, using unrelated piles of synthetic and real images rather than matched pairs. Applied to video, this fails in two ways at once. If each frame is repainted independently, textures flicker and the same car or pedestrian looks different from every angle. If each frame is repainted conditioned on the previous repainted frame, small errors are fed back as context and compound until structures "melt", which is the temporal drift seen in autoregressive video generators. A third failure is borrowed from text-to-3D work: real photo collections are dominated by front-facing views, so a translator asked to make the back of a synthetic head or the rear of a car "look real" tends to paint on a face or a front grille. This is the Janus problem.
This note proposes to stop treating the previous output frame as the source of consistency and to use the simulator's own geometry instead. The simulator knows exactly which surface point each pixel shows, including on moving and deforming objects. We translate each surface point once, from the best available view, with a translator that is told which part of the object it is looking at. We store that appearance in a memory indexed by surface coordinates, and every later frame reads from the memory through exact projection. Because no stored value is ever computed from a frame that was itself conditioned on the memory, the feedback loop that causes drift is cut. Because the write view and part label are chosen from simulator ground truth, the front-view bias behind the Janus problem has no rear views to act on.
Core Claim
The central claim is that, in simulation, the three failures (flicker across views, drift over time, and Janus duplication) share one cause and one remedy. The cause is that consistency is inferred from generated pixels. The remedy is to anchor appearance to ground-truth surface correspondence, which simulators provide for free.
Prior work motivates this from both sides. On the 3D side, Carve3D measures and rewards multi-view consistency by lifting generated views into a radiance field with a feed-forward reconstruction model, re-rendering them, and comparing with LPIPS. DreamCS reduces Janus artifacts through a learned 3D reward and differentiable meshing, at roughly 30% extra training time per iteration. Both must estimate geometry because they have none. On the video side, 3DScenePrompt shows that projecting an explicit 3D memory into target views as conditioning cuts multi-view misalignment sharply. It builds that memory with dynamic SLAM and masks out moving objects, so its own stated limitation is that dynamic foregrounds still drift. The sim-to-real setting removes both obstacles: geometry is exact, and dynamic or non-rigid objects come with their own canonical coordinates through rigging and skinning. That makes possible what these papers could not do, namely a surface memory that covers moving people and vehicles.
The drift argument is short. Causal State Compression for Long-Horizon Video World Models shows that conditioning on one's own past outputs, with Lipschitz constant $L>1$ and one-step error $\epsilon$, gives error that grows like $O(L^T\epsilon)$ over horizon $T$. In our design a texel's stored value comes from a single translator pass that is not conditioned on the memory. Each rendered frame then reads stored values and applies a bounded refinement. The per-pixel error therefore satisfies
$$e_t \le \epsilon_w + \epsilon_r,$$
where $\epsilon_w$ is the error of the one-time write and $\epsilon_r$ is the error of the read-and-refine step. The bound does not depend on $T$, because neither term is ever fed back into later writes. This is a structural guarantee, not a learned one. Its cost is that appearance cannot change unless we allow it to: lighting changes, mud, and wear need a separate, explicitly modeled channel. We treat that as part of the research, not a footnote.
Research Program
The system has three parts.
The translator. This is an image diffusion model, a generator that turns noise into images by gradual denoising. It is adapted from synthetic to real without paired data, using the round-trip cycle constraint surveyed in Generative Adversarial Networks: Recent Developments. We reuse that technique as-is. The translator is conditioned on simulator depth, normals, and segmentation. It differs from standard unpaired translators in one input: a per-pixel part label and object-relative viewing angle, such as "back of head, viewed from 160 degrees." This targets the mechanism that DreamBooth3D and Understanding Pure CLIP Guidance for Voxel Grid NeRF Models identify behind Janus artifacts: priors that judge each view in isolation pull toward the canonical front. Telling the translator which side it sees should remove that pull. Condition Matters in Full-head 3D GANs finds the same thing for 360-degree head generation, where explicit conditioning prevents collapse to frontal features.
The surface memory. Each static mesh and each rigged dynamic asset gets a texture-space appearance store, conceptually similar to the pose-aware UV mapping in Dream, Lift, Animate. Texels start empty. When a texel is first seen from a view whose angle to the surface normal falls below a threshold, and the translator's confidence passes a gate, its value is written and frozen. We will study the confidence signal; disagreement across a few noise seeds is the first candidate, motivated by the seed divergence analysis in In-N-Out. Texels seen only at grazing angles are written provisionally and can be overwritten once by a better view, so the number of writes per texel is bounded.
The renderer and refiner. Each output frame is first rendered from memory via simulator projection. A lightweight refinement pass then fills disocclusions, adds view-dependent effects such as specular highlights, and harmonizes lighting. This pass is the same translator, run from a partially noised version of the memory render. Its output is never written back.
Two open problems become explicit studies. The first is the appearance-change channel: a low-dimensional per-object lighting and weather code, updated over time but constrained to globally smooth changes so it cannot carry texture. The second is the tradeoff between detail and rigidity flagged in the 3D-consistency literature. We will measure how much high-frequency detail the frozen memory loses compared with per-frame translation.
Baselines
The naive reference point is training downstream models on raw simulator frames with no translation. Every translation method has to beat this on the real-domain task, or the exercise is pointless.
The first translation baseline is a per-frame unpaired translator. It uses the same diffusion backbone, the same cycle-consistent adaptation, the same depth and segmentation conditioning, and the same real image set, but no memory and no part or view input. It isolates what the memory and conditioning add. It should look sharpest on single frames and worst on cross-view and revisit consistency.
The second is an autoregressive video translator. It conditions on its previous outputs and is trained with self-rollouts, following Self Forcing. Self Forcing trains on its own unrolled generations with distribution-matching losses and a rolling cache, and it reports 84.31 on VBench with Wan2.1-T2V-1.3B. This is the strongest existing method for controlling drift. Ours must show flat error over horizons beyond its training context, where Self Forcing's fidelity is known to decay.
The third is a static-only memory in the style of 3DScenePrompt. It uses our pipeline, but only background geometry enters the memory and dynamic objects are translated autoregressively. 3DScenePrompt reports MEt3R of 0.041 against 0.181 for its baseline on its own camera-controlled generation benchmark. That number is not transferable to sim-to-real translation, and we found no reported result on Lacuna for any method on temporally consistent sim-to-real video translation. Every number in our comparison will therefore be produced by us under matched conditions. Beating this baseline means a lower Janus rate, lower revisit error, and lower MEt3R on dynamic objects specifically.
The fourth is a correspondence-reward fine-tuned translator. It is the per-frame translator fine-tuned with on-policy RL as in Carve3D, but the reward is computed from exact simulator correspondences instead of an LRM reconstruction. Carve3D reports an MRC of 0.0606 against 0.0892 for its base model on 415 DreamFusion prompts. This is the strongest learned-consistency competitor and is described below.
Two ablations isolate the key ingredient. In the first, the memory is updated as a running average of refined outputs, which restores the feedback loop. In the second, the part and view input is removed.
All methods share the backbone, the real image corpus, the simulator episodes, and the GPU-hour budget for adaptation and tuning. Hyperparameter search gets equal trials per method.
Why not just…?
Why not just warp the previous translated frame with the simulator's exact optical flow and penalize differences? This is the most obvious alternative and a real one for short clips. Flow warping only links consecutive frames, so a surface that leaves view and returns has no constraint tying its two appearances together. Small per-step mismatches also chain multiplicatively, which is the $O(L^T\epsilon)$ regime. We include it as a variant of the autoregressive baseline. The loop-closure experiment below is designed to separate it from our method. If flow warping matches us on revisit error, the memory is unnecessary and we will say so.
Why not just fine-tune a per-frame translator with a consistency reward, as Carve3D does? With simulator correspondences the reward is cheap, so the overhead objection to Carve3D does not apply, and this is the alternative most likely to compete. Its weakness is that it makes consistency more likely without guaranteeing it. Each frame is still sampled from fresh noise, so inconsistency shrinks but never reaches zero, and nothing forbids a front-facing hallucination on a rear view that happens to be internally consistent. Our experiments report the full distribution of revisit error, not just its mean, along with Janus rates. If the reward-tuned model matches us on both, the right conclusion is that simulator correspondence is the ingredient and the memory is optional. We would report that.
Why not just texture the simulator assets offline once with a 2D prior and skip per-frame translation? Scoring rendered views with a 2D prior and optimizing a texture against those scores is exactly the setting where DreamCS documents Janus artifacts in 52% of MVDream assets. It also gives up per-frame refinement for lighting and disocclusion. Our write-once memory can be seen as an online, view-gated version of offline texturing. The no-refinement ablation tests whether the per-frame refiner earns its cost.
Proposed Experiments
Episodes come from a driving or robotics simulator that exports depth, normals, instance and part segmentation, and texture-space coordinates for rigged pedestrians and vehicles. The target domain is an unpaired corpus of real camera footage from a comparable setting. Real-domain test sets with labels are used only for downstream evaluation. Lacuna listed no specific real driving benchmark, so the choice of corpus is a pre-registered decision, not one tuned for results.
The first decisive check is loop closure. We script camera trajectories that leave and later revisit the same surfaces, and pedestrians that turn a full circle. Because the simulator gives exact correspondence, we can compute revisit error: the perceptual distance between a surface point's translated appearance on first and later visits. We plot it against elapsed time out to horizons several times longer than any training context. The prediction is a flat curve for our method and growing curves for the autoregressive and flow-warping baselines. If ours also grows, the write-once argument has failed in practice, most likely through the refiner, and the running-average ablation will show where.
The second check is the Janus rate on rear and side views of people, animals, and vehicles. Rates are judged blind by annotators, following DreamCS's convention of reporting the fraction of affected assets. We compare the full system, the system without part and view input, and the reward-tuned translator. We also report MEt3R and an MRC-style reconstruction consistency score, so results sit on the same scales as 3DScenePrompt and Carve3D.
The third check is the reason sim-to-real translation exists. We train the same detector and segmenter on raw, per-frame-translated, autoregressively translated, reward-tuned, and memory-translated simulator data, then evaluate on held-out labeled real video. We report frame-level accuracy and accuracy on tracking-style temporal tasks, where flicker in training data should hurt most.
We will additionally report single-frame realism, measured as a feature-distribution distance to real frames. This makes visible the detail cost of freezing appearance, against the consistency and downstream gains it buys.
References
- Carve3D: Improving Multi-view Reconstruction Consistency for Diffusion Models with RL Finetuning
- DreamCS: Geometry-Aware Text-to-3D Generation with Unpaired 3D Reward Supervision
- 3D Scene Prompting for Scene-Consistent Camera-Controllable Video Generation
- Causal State Compression for Long-Horizon Video World Models: A Bounded-Drift Theory and Efficient Architecture
- Generative Adversarial Networks: Recent Developments
- DreamBooth3D: Subject-Driven Text-to-3D Generation
- UNDERSTANDING PURE CLIP GUIDANCE FOR VOXEL GRID NERF MODELS
- Condition Matters in Full-head 3D GANs
- Dream, Lift, Animate: From Single Images to Animatable Gaussian Avatars
- In-N-Out: Lifting 2D Diffusion Prior for 3D Object Removal via Tuning-Free Latents Alignment
- Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion