Sim-to-Real Physics as a Compositional Generalization Problem

Video generators trained on internet footage produce beautiful clips in which objects float, pass through each other, or lose mass. One fix that works is to fine-tune them on simulated physics. Simulation covers every drop height, material, and bounce we care about, but it looks synthetic. Real footage looks right but covers only a few physical situations. The hope is that physics learned from simulated videos carries over to real-looking videos. This note argues that the hope rests on a combinatorial bet. In the training data, "real appearance" has only ever co-occurred with a handful of physical regimes. We then ask the model to render a combination it has never seen: real appearance paired with physics it saw only in simulation. Seen this way, the sim-to-real gap for video physics is a held-out-combination problem, and tools built for compositional generalization should apply to it directly.

Concretely, we propose to build the generator's conditioning so that "rendering domain" (simulated vs. real) is one independent factor and each physical property (drop height, bounciness, camera distance) is another. Each factor gets its own small additive control head on a shared backbone. We then test whether physics learned in simulation transfers to real-looking video for exactly the physical regimes that real data never covered. If it does, simulation coverage becomes something we can spend on real-world physics. The design also offers a direct explanation of, and possible remedy for, the "visual quality tax" seen when physics fine-tuning blurs real textures.

Core Claim

Our claim is that a video diffusion model will transfer physics from simulation to real appearance far better on unseen (real × physical-regime) combinations if two conditions hold. First, the rendering domain and each physical factor must enter through separate additive heads rather than a single entangled conditioning vector. Second, the physics should then stay measurably unchanged when only the domain head is switched. The method we reuse comes from Compositional Flow Matching with Factored Velocity Fields. That work splits a flow-matching generator's velocity field (the learned direction that moves noise toward an image) into a shared base network plus per-factor modulation heads summed at the bottleneck. On MPI3D-real with half of the factor combinations held out, it raised held-out attribute accuracy from about 47% to about 78%. That result is static, at 64×64 resolution, and has no notion of time. We carry the same structure into a video model and assign it a new role: one of the factors is the sim/real domain itself. Written out, the per-step velocity is

$$v(x_t, t \mid c) = v_{\text{base}}(x_t, t) + \delta_{\text{dom}}(x_t, t; d) + \sum_{k=1}^{K} \delta_k(x_t, t; c_k),$$

where $x_t$ is the noisy video latent at diffusion time $t$, $d \in \{\text{sim}, \text{real}\}$ is the rendering domain, and $c_k$ is the value of physical factor $k$.

The motivating evidence is PISA Experiments. It post-trained Open-Sora v1.2 on 5,000 Kubric drop simulations and then optimized rewards for segmentation, optical flow, and depth consistency. The result beat Sora and Runway Gen3 on real drop trajectories. Yet it fails when drop height or camera depth leaves the training range, and it pays a visual quality tax in the form of blur and ghosting. Our reading is that both failures are entanglement. Physics knowledge and simulated appearance are learned in the same parameters, so pulling toward simulated physics also pulls toward simulated texture, and the real-appearance mode never learns the extra regimes.

Research Program

The data design is the heart of the project. Using Kubric with scanned household objects, as PISA did, we render drop videos over a grid of three physical factors. Drop height has three bins. Restitution class has two values: objects that settle and objects that bounce. Camera distance has three bins. Simulation covers all 18 physics cells.

For the real domain we start from the 361 high-frame-rate PisaBench Real clips and annotate height and distance bins. Since that set is small and is also the evaluation pool, we additionally record roughly 600 new 120 fps phone videos with tape-measured heights and known bouncy and non-bouncy objects. Each real clip is assigned to its cell. During training the model sees real footage from only a chosen subset of cells, and we vary the held-out fraction across 25%, 50%, and 75%, mirroring the splits of the factored flow-matching paper. Test clips are always real footage from held-out cells. Collecting new real data is a genuine cost and a limitation: the approach needs coarse factor labels on real data, which the factored method has always required.

The model is Open-Sora v1.2. Its existing timestep and text modulation is extended with one small head for the domain and one per physical factor, and their outputs are summed into the modulation signal at each transformer block. We train it in PISA's two stages: supervised fine-tuning on the mixed sim and real data, then reward optimization with PISA's object rewards applied to simulated samples only, where ground-truth masks exist. Under the factored design, reward gradients can be restricted to the base network and the physical heads, while the domain head is trained only by the denoising loss on real clips. This restriction is the mechanism by which we expect to avoid the quality tax.

We also add a diagnostic that does not exist in prior work, which we call domain-swap invariance. For any physics condition, we generate with $d=\text{sim}$ and with $d=\text{real}$ from the same noise. We track the object in both outputs and report the mean trajectory distance between them. If this number is small, physics lives outside the domain head. If it is large, appearance and dynamics are tangled together. Monolithic models can be measured the same way by swapping their domain token.

Baselines

The naive reference point is Open-Sora v1.2 with no physics post-training, prompted with the drop description. Alongside it we report the proprietary numbers PISA published, so readers can see the scale of the problem.

The strongest existing method is PISA itself. The best reported figure is a real-world trajectory L2 error of 0.067 on PisaBench Real, against 0.174 for Sora and 0.187 for Runway Gen3 (PISA Experiments). That number comes from the full real distribution rather than held-out cells, so we rerun PISA's pipeline on our identical sim-plus-real training data, with the same reward optimization stage, the same number of steps, and the same hyperparameter search budget. Beating it means lower trajectory L2 on real held-out cells at matched visual quality, with the difference holding across three seeds.

The decisive ablation keeps the same backbone, data, and compute but feeds domain and physics in through one monolithic conditioning vector, as a concatenated label embedding. This model holds exactly the same information as ours and differs only in factorization, so any gap is attributable to the additive structure. A second ablation keeps the factored heads but lets reward gradients reach the domain head, which isolates the gradient-routing claim about the quality tax. We also run Slice-CFG-style test-time composition on the monolithic model, because the factored flow-matching paper compared against exactly that and found architecture beat sampling. If our factored model does not exceed the monolithic ablation on held-out real cells by a margin larger than seed variance, the core claim fails.

Why not just…?

Why not just simulate everything and let real data handle appearance? That is precisely the setting we test. Simulation does cover every cell, and the open question is whether that coverage reaches the real-appearance mode. PISA already trains on broad simulation plus implicitly real-like pretraining and still breaks at out-of-range heights and depths. The PISA-rerun baseline is this alternative in its strongest form, so if it matches our model on held-out real cells, we will report that factorization is unnecessary.

Why not just make the simulator photorealistic so there is no domain factor? Ray-traced rendering does narrow gaps: RAFT pre-trained on Kubric reached 4.82 endpoint error on KITTI-15, against 7.63 from FlyingChairs (Kubric). But that optical-flow generation cost about three CPU-years, and some residual gap remains. This is a real threat. We include an arm that re-renders the simulated set at maximum Cycles quality and trains PISA on it. Photorealism and factorization are complementary, and the measurement that settles the question is whether the factored model's advantage shrinks to zero under better rendering.

Why not just put a physics engine in the loop, as PhysGen does? PhysGen simulates 2D rigid-body motion from parameters estimated by GPT-4V and uses diffusion only to render the result. It reached 4.14/5 human realism. However, its physics lives in the image plane and fails on motion toward or away from the camera, which is exactly our camera-distance factor. On a planar-motion subset we report it as an in-scope upper bound, and we expect it to lose on out-of-plane cells.

Why not just add more diverse real footage? Are Object-Centric Representations Better at Compositional Generalization? found that dense models close the gap with structured ones when training diversity is high, through interpolation rather than composition. We therefore expect the factored advantage to shrink as real coverage grows. The 25/50/75% holdout sweep measures this curve directly. The proposal is worth pursuing only if the advantage is substantial at the 50% and 75% holdouts, where collecting real footage of every regime is impractical.

Proposed Experiments and Decisive Checks

The primary metric is PisaBench trajectory L2 together with its Chamfer distance and object-permanence IoU, computed on real held-out cells. We add a restitution check: the ratio of first bounce height to drop height, read from tracked trajectories and compared with the same objects in reality. Visual quality is measured by FVD against held-out real clips, so that the quality tax is quantified rather than eyeballed. Violation rates for impenetrability and mass conservation come from the VILA-2B judge of WorldModelBench; the judge is trained on 67K human labels and is reported as more accurate than GPT-4o.

There are three decisive checks. First, the factored model must beat the monolithic ablation on held-out real cells at the 50% holdout. Second, its domain-swap trajectory distance must be markedly lower than the monolithic model's, which would show that the gain comes from separating physics from appearance. Third, restricting reward gradients away from the domain head must reduce FVD degradation relative to PISA without raising trajectory error.

A secondary probe touches the "reality bias" reported in PhyWorldBench, where models asked for impossible physics produce static scenes. We add a reduced-gravity cell that exists only in simulation and ask whether the real-appearance mode can render counterfactual physics when that physics is a separate head rather than a prompt. If all three checks fail, the conclusion that the sim-to-real physics gap is not about combinations, and is instead driven by low-level dynamics mismatch, is itself worth publishing.