Is "Hallucinated Physics" Really Geometric Inconsistency? Simulator-Anchored Consistency for Multi-View Video Augmentation in Robot Learning

Robot learning teams increasingly use video generators to repaint simulator footage so it looks like real camera footage, then train manipulation policies on the result. The approach works up to a point and then stops working. RoboTransfer shows that mixing generated and real demonstrations 1:1 raises real-robot success sharply. It also shows that pushing the synthetic share above 50% makes policies worse, and its authors attribute this to the generator teaching "hallucinated physics." We propose a narrower and testable explanation. A generator that repaints a simulated scene does not change the physics, because the simulator already fixed the object trajectories. What it can change is geometry. An object drawn a few pixels off in the wrist camera relative to the overhead camera is the robot-video version of the Janus problem, where a generated 3D object shows contradictory features from different viewpoints. Edges that drift from frame to frame are inter-frame inconsistency. Both kinds of error hand the policy action labels that disagree with what it sees. Real cameras never produce either error, so they are also a domain gap of their own.

Our approach rests on one fact: the simulator knows, for every pixel, which physical surface point it shows, in every camera and every frame. That gives exact dense correspondences across views and across time for free, including for objects that move and touch each other. Prior consistency methods have to estimate these correspondences, or they work only on static scenes. We use the simulator's correspondences in two ways. During training, they become a supervision signal on the generator's internal features. Afterwards, they become an audit that measures how far each generated clip departs from the true scene geometry. The research question is whether removing geometric inconsistency, and nothing else, lets synthetic data keep helping past the 50% ceiling.

Core Claim

In simulator-conditioned video augmentation, most of what is called hallucinated physics shows up as measurable disagreement between generated pixels and the simulator's geometry. This disagreement appears both across cameras and across time, and it is what makes heavily synthetic training sets hurt policies. Two consequences follow. First, a per-clip consistency error computed against simulator ground truth should predict how much a policy degrades as the synthetic share grows. Second, a generator trained with simulator-supplied correspondence supervision should keep the policy improving, or at least flat, beyond a 50% synthetic mix, at a point where a RoboTransfer-style generator declines. If consistency error does not predict degradation, the claim is wrong and the physics explanation stands. That outcome would also be worth knowing.

Relation to Prior Work

Three lines of work motivate the proposal. The 4D generation literature shows that cross-view and cross-frame errors become geometric artifacts once someone fits a 3D model to them. Consistent4D regularizes a dynamic radiance field with an interpolation-based temporal loss. EG4D adds a per-frame color transform so that 4D Gaussians do not deform to explain flicker. FB-4D carries attention features across frames. All three produce graphics assets and stop there; we ask what the same errors do to a downstream policy.

Robot augmentation work supplies the setting and the open puzzle. M3A Policy repaints object materials in monocular demonstrations. SimGen turns simulator layouts into realistic single-camera driving images. RoboTransfer handles multiple cameras by placing them side by side on one wide canvas and letting self-attention discover the geometry from data. None of these measures geometric faithfulness to the conditioning state. EnerVerse generates multi-view embodied rollouts, but the file gives no methodological detail, so we treat it only as context.

The method itself is borrowed. ViCoDR trains a video diffusion model with a ranking loss that makes internal features agree at corresponding 3D points. It reports an 11% improvement in MEt3R (a 3D reprojection-consistency metric) on RealEstate10K. Its limits are that it estimates correspondences with a pretrained geometry model and evaluates on static scenes. Our change is to take correspondences from the simulator, which makes them exact, and to apply the loss across cameras and time in scenes with moving and contacting objects.

The most direct architectural competitor is explicit epipolar attention. Collaborative Video Diffusion and EpiDiff restrict cross-view attention to geometrically valid lines. This idea constrains the design. Epipolar constraints relate only views captured at the same instant, so they cannot catch drift across time. EpiDiff is also reported to degrade on distant views, and a wrist camera and an overhead camera are very far apart.

Research Program

We build a simulator-conditioned multi-view video generator. It starts from a fine-tuned Stable Video Diffusion backbone and uses a RoboTransfer-style setup: multi-camera canvas input, with depth and normal conditioning rendered by the simulator rather than estimated. We add one ingredient on top.

From the simulator's depth buffers, camera poses, and per-object poses, we compute dense pairs of pixels $(p, q)$. Each pair is the same visible surface point seen in view $a$ at frame $t$ and in view $b$ at frame $t'$. Pairs cover the same view at a different time, a different view at the same time, and a different view at a different time. The correspondence loss on the generator's intermediate features $f$ is a margin ranking loss:

$$ \mathcal{L}_{\text{corr}} = \sum_{(p,q)} \max\big(0,\; m + d(f(p), f(q)) - d(f(p), f(n))\big), $$

Here $d$ is cosine distance, $n$ is a hard negative sampled near object edges, and $m$ is a margin. This is ViCoDR's ranking-and-edge-sampling design, supplied with exact pairs. The term is added to the usual denoising loss. It adds no cost at generation time.

The second component is an audit. For each generated clip, an off-the-shelf matcher and tracker estimate correspondences on the generated frames, and we compare them with the simulator's true correspondences. Separately, we compare masks segmented from the generated frames with the simulator's object masks. Together these give a cross-view error, a temporal error, and a "contact faithfulness" error. The last one flags generated penetration or floating, which is exactly what RoboTransfer calls hallucinated physics. The audit serves both as the diagnostic for the core claim and as a cheap filter that can be compared against the training-time loss.

Baselines

The naive reference point is a policy trained only on real demonstrations, with the same Diffusion Policy architecture and the same real data budget. A second reference is a policy trained on raw simulator renders with standard texture and lighting randomization. This represents classical sim-to-real transfer, for which ManiSkill2 reports 91% PickCube success in simulation and 60% on a physical robot without real fine-tuning.

The strongest existing method is RoboTransfer. On its unseen-object-and-environment setting it reports 46.7% real success with its generated data against 13.3% real-only, and it identifies a 50/50 mix as optimal. Its code and robot setup are not available to us, so we will reimplement it on our hardware and our tasks. We will use the identical backbone, identical simulator trajectories, identical generator fine-tuning steps and resolution, and the same policy training budget. The only differences will be our correspondence loss and our audit. Our reproduction must recover a real-plus-synthetic gain over real-only of similar shape before any comparison counts. Beating it means two things: equal or better success at the 50/50 mix, and a statistically clear advantage at 75% and 100% synthetic, measured over repeated real trials per condition.

A third baseline replaces the canvas self-attention with Collaborative Video Diffusion-style epipolar attention computed from simulator poses. It isolates whether an architectural constraint on views at the same instant matches a training loss that also spans time.

Ablations isolate the key ingredient. We will use only same-time cross-view pairs, then only same-camera temporal pairs, and then both. We will also add EG4D's per-frame affine color correction to the RoboTransfer reimplementation, which tests whether removing flicker alone explains any gain.

Why not just…?

Why not just randomize simulator textures instead of using a generator at all? Randomized rendering is consistent by construction, so it is the cleanest control for the consistency hypothesis. Its weakness is realism. The ManiSkill2 drop from 91% to 60% shows the visual gap it leaves, and RoboTransfer's gains over real-only come from realism a randomizer cannot supply. This baseline is still a real threat at high synthetic ratios, because it never hallucinates. If it matches our method at 100% synthetic, consistency is necessary but realism adds nothing beyond it. We would report that outcome.

Why not just filter bad clips with the audit and skip retraining the generator? This is the most serious alternative, and we run it directly. If filtering RoboTransfer outputs by the simulator-anchored consistency score recovers most of the gain, the contribution reduces to the diagnostic and the filter. That is cheaper and still useful. We expect filtering to discard many clips, because inconsistency is common in canvas generation, so we report policy success against the number of clips kept as well as against the raw synthetic share.

Why not just add epipolar attention? Epipolar geometry relates only frames captured at the same instant. It says nothing about texture drift across time and nothing about an object being drawn penetrating a gripper the simulator says it is touching. EpiDiff's known degradation on distant views also bears on the wide wrist-to-overhead baseline. The epipolar baseline and the temporal-only ablation together show which constraint matters.

Why not just blame physics and add a physics engine? In our setting the physics engine already exists, because trajectories come from simulation. Any physical implausibility therefore has to appear as disagreement with the simulator's masks and depth, and the contact-faithfulness audit measures that disagreement directly.

Proposed Experiments

We generate demonstrations from RLBench and ManiSkill2 tasks with multi-camera rigs: wrist plus third-person views, with RLBench's automated waypoints providing unlimited trajectories. Real evaluation mirrors RoboTransfer's tasks (spoon pick-and-place and towel folding) and adds ManiSkill2-style PickCube, with unseen objects and backgrounds. Generator quality is measured by FVD, by MEt3R, and by the three simulator-anchored errors. Policy quality is real-robot success.

The first decisive check is correlational. Across many generator variants and checkpoints, we test whether clip-level consistency error predicts the slope of policy performance between 50% and 100% synthetic. A second, controlled test runs entirely in simulation. We take perfectly consistent randomized renders and inject calibrated cross-view object offsets and temporal jitter, with no generator involved. If the over-50% collapse appears from injected inconsistency alone, the geometric explanation gains strong support. If it does not appear, the explanation is weakened.

The second decisive check is the ratio sweep at 25, 50, 75, and 100% synthetic, comparing our generator, the RoboTransfer reimplementation, the epipolar variant, audit filtering, and the ablations. The proposal succeeds if the correspondence-trained generator removes the decline past 50% while matching RoboTransfer at 50/50. It fails if consistency errors fall but the decline persists, which would point back to non-geometric causes.