Video world models generate the future one chunk at a time and feed their own output back in as input. Over long rollouts the output degrades: textures morph, walls bend, colors wander. This is called temporal drift. We argue that a large and identifiable share of this drift starts in one kind of pixel: surfaces that were hidden and become visible, either because the camera moved or because a foreground object moved out of the way. When a surface is first revealed (a disocclusion), the model has no evidence for it and must invent it. Current models then go on reinventing that surface at every later step, because nothing marks it as already decided. Our proposal attaches a provenance tag to each pixel that records whether it was observed or invented. Each revealed region is invented exactly once. After that, a coarse-to-fine optical flow estimator, which predicts where each pixel moves by refining a coarse motion guess at successively finer resolutions, only moves the region. The generator never repaints it.
The approach runs at inference time and keeps the generator frozen. Its main motion component already transfers from synthetic training data to real video. That property lets us study a question the literature has not asked: whether drift that is attributed to disocclusions in simulation, where exact visibility masks exist, shows the same pattern in matched real footage. Virtual KITTI 2 provides synthetic clones of real KITTI sequences, which makes this comparison possible.
Core Claim
The claim is that per-pixel provenance, combined with a commit-once rule for invented content, changes how error grows in revealed regions. Without it, those regions are regenerated at every step, so errors compound the way generated frames compound when fed back. With it, the remaining error in those regions comes only from transport, meaning errors in moving content, which are bounded by optical flow accuracy. The prediction is testable. Under the commit-once rule, error should grow slowly in regions revealed early in the rollout. Without the rule, those same regions should get worse at roughly the rate of fully generated content.
The work in the file motivates this claim from three directions without testing it. 4DGS-WAM (4DGS-WAM) shows that moving explicit 3D primitives, rather than regenerating pixels, stops identity and texture drift. However, it cannot complete newly revealed background and falls back on an ad-hoc "frozen fusion" patch. DynaMem (DynaMem) anchors color to frame 0, which permanently propagates any early error and does nothing specific for content revealed later. CamGeo (CamGeo) reports "content collapse" when the camera moves into regions not visible in frame 0. Each of these points to disocclusions as the place where drift enters. None of them track where invented content came from or keep it stable after it is created.
Research Program
The system maintains a canvas: the current frame plus, for every pixel, its provenance (observed, transported, or invented), the step at which it was last invented, and a confidence score. Each step has four parts.
- Draft. A frozen driving world model produces a draft of the next frame. We use Vista (Vista), a Stable Video Diffusion model that conditions on the past three frames to give it motion priors.
- Motion. WAFT (WAFT) estimates flow from the canvas to the draft. It refines motion by warping features at half resolution and was trained only on synthetic data. The file reports that it has the best zero-shot KITTI transfer.
- Reveal detection. The canvas is forward-warped along this flow. Pixels with no incoming source are marked as reveals. Pixels whose coarse and fine flow estimates disagree are also marked as uncertain and treated as reveals.
- Re-denoising. The draft is partially re-noised and denoised again. Transported pixels are held fixed to their warped canvas values, so the generator may only write into revealed and uncertain pixels. The newly written pixels are then committed to the canvas as invented content.
The coarse-versus-fine disagreement test is where hierarchical motion estimation contributes. Coarse-to-fine estimators are known to lose thin or fast structures at the coarse level, as described for PWC-Net (PWC-Net). They are also known to fail under extreme displacement and occlusion, as reported for WAFT. If such pixels were simply transported, they would leave "ghost" copies of the object. Releasing them to the generator avoids this. We treat the disagreement test as an assumption to be tested, not as the contribution. If a plain forward-backward consistency check performs equally well, we will use that instead.
Synthetic-to-real transfer appears in two places.
- Training the reveal detector. A small model that predicts visibility, deciding whether a pixel's warped value can be trusted, is trained on Virtual KITTI 2 (Virtual KITTI 2). This dataset provides exact forward and backward flow and scene flow, so true disocclusion masks can be computed for free. This follows the finding in the amodal segmentation work (EUREKA / IRAIS) that synthetic geometric supervision teaches occlusion reasoning that transfers to real images. That finding comes from single frames and rigid retail objects, and the same paper reports failures on non-rigid objects. We therefore evaluate pedestrians separately.
- Diagnosis. Running the same rollouts on Virtual KITTI 2 clones and on their real KITTI originals tests whether the breakdown of drift into revealed versus never-hidden regions holds across domains. If it does, simulation becomes a reliable tool for diagnosing drift.
Baselines
Unmodified Vista rollout (naive reference). This is the frozen backbone generating autoregressively with no canvas. Every method below uses the same weights, the same conditioning frames, the same actions, and the same number of denoising steps. All extra inference passes are counted in a reported compute-per-frame figure. The file gives Vista's advantage over Drive-WM only in relative terms (55% better FID, 27% better FVD) and gives no absolute values. We will therefore measure this baseline ourselves.
Per-frame inpainting. This uses the same warp and the same reveal masks, but holes are re-inpainted at every step and nothing is committed. It isolates the commit-once rule, which is the ingredient that differs from Beyond the Visible (Beyond the Visible). That work documents flicker and ghosting from stochastic per-frame filling.
DynaMem on the same backbone. DynaMem (DynaMem) is a training-free method that combines hierarchical frame memory with color anchoring to frame 0. It is the strongest drift fix that also leaves the backbone frozen. It reports VBench Subject Consistency of 0.9677.
4DGS-WAM (strongest explicit-transport competitor). 4DGS-WAM (4DGS-WAM) reports future-prediction PSNR 18.80 and SSIM 0.596 on KITTI-MOT, compared with 17.46 and 0.482 for Epona. We follow its KITTI-MOT protocol exactly. We also evaluate a strengthened version in which its frozen-fusion patch is replaced by the same inpainter we use, with the inpainted content committed to its background Gaussian bank. We count our method as beating it if we exceed 18.80 PSNR at the matched horizon, and also show lower error inside revealed regions at longer horizons.
Ablations of the key ingredients.
- Reveal masks from forward-backward checks only, versus masks from coarse-fine disagreement.
- WAFT versus RAFT (RAFT) as the transport model.
- A reveal detector trained on synthetic data versus a hand-thresholded one.
Why not just…?
Why not just give 4DGS-WAM a good inpainter? This is the most serious threat. It would make "invent once, then transport" a property of the 3D Gaussian background bank. We expect our approach to keep two advantages:
- It needs no per-scene 3D optimization.
- It handles non-rigid revealed content in 2D, whereas 4DGS-WAM moves dynamics at a single object level.
The strengthened 4DGS-WAM baseline settles this directly. If it matches us on revealed-region error, the contribution reduces to the provenance diagnosis and the sim-to-real attribution study, and we will say so.
Why not just retrieve past frames, as VRAG does? VRAG (VRAG) resets drift when the camera returns to a previously seen pose, reaching SSIM 0.9116 against 0.4436 on RealEstate10K. Retrieval cannot help with a surface revealed for the first time, because nothing has been stored for it. Forward driving consists almost entirely of first-time reveals. We will report results split into revisit and first-reveal segments, which separates the two mechanisms.
Why not just fine-tune the world model on real driving video? Fine-tuning does not mark invented content, so the model would still repaint it at each step. AnyCamVLA (AnyCamVLA) also reports that fine-tuning causes forgetting. Fine-tuning and our method are compatible, so if compute allows we will apply the canvas to a fine-tuned backbone as well.
Why not just use a forward-backward flow check for reveals? This may well be sufficient. The mask ablation decides it, and we will adopt the simpler mask if it matches.
Proposed Experiments
Region-split rollouts. We will run rollouts of 8 to 64 frames on Virtual KITTI 2, on its real KITTI clones, and on nuScenes. Every evaluation pixel is assigned to one of three groups: never hidden, revealed early, or revealed late. On Virtual KITTI 2 the groups come from exact masks. On real footage they come from WAFT forward-backward checks on the ground-truth future frames. For each group we report PSNR and LPIPS against ground truth over the horizon, along with FVD.
Re-invention rate. For each revealed pixel, we measure how much its appearance changes after it is revealed, after compensating for motion using ground-truth or pseudo-ground-truth flow. This metric requires no knowledge of what the hidden surface "should" look like, which cannot be known.
Decisive checks.
- Under the commit-once rule, error in early-revealed regions should rise much more slowly with horizon than error in the per-frame inpainting baseline. If the two error curves are parallel, the claim is false.
- The share of total drift attributed to revealed regions on Virtual KITTI 2 should match that share on the real KITTI clones within confidence intervals. If it does not, simulation cannot be used to diagnose this failure.
- We report all results separately for pedestrian regions, where the occlusion priors learned from synthetic data are expected to transfer least well.
References
- 4DGS-WAM: Bridging Past and Future with an Object-Centric World Action Model based on 4D Gaussian Splatting
- DynaMem: Consistent Long Video Generation via Hierarchical Memory and Motion Priors
- CamGeo: Sparse Camera-Conditioned Image-to-Video Generation with 3D Geometry Priors
- Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability
- WAFT: Warping-Alone Field Transforms for Optical Flow
- PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume
- Virtual KITTI 2
- Amodal Instance Segmentation with IRAIS Dataset for Sim-to-Real Transfer
- Beyond the Visible: Disocclusion-Aware Editing via Proxy Dynamic Graphs
- RAFT: Recurrent All-Pairs Field Transforms for Optical Flow
- Learning World Models for Interactive Video Generation
- AnyCamVLA: Zero-Shot Camera Adaptation for Viewpoint Robust Vision-Language-Action Models