Counterfactual Training Data for Feed-Forward 4D Reconstruction That Does Not Mistake Lighting for Motion

Recent networks can turn a phone video of a moving scene into a 3D model in under a second. They do this in a single forward pass, with no per-video optimization. They learn this skill almost entirely from synthetic videos, and those videos contain a narrow set of motions. They also almost never show a surface whose color changes while it stays put: a shadow sweeping across a wall, a flickering lamp, a highlight sliding over a car. Such a network has no reason to tell "the pixel changed because the surface moved" apart from "the pixel changed because the light changed," so it can report shadows and highlights as motion.

We propose to fix this mainly with data rather than architecture. We will build procedurally generated training videos in matched pairs. In one member of each pair the geometry moves and the appearance is fixed. In the other the appearance changes and the geometry is fixed, and the true motion is zero. We will also draw motion from many distinct motion-generating programs rather than a few. The claim is that a 4D reconstructor trained this way will separate motion from appearance better than architectural tricks do, and will generalize further to real motion. We test both parts of the claim on real monocular video benchmarks.

Core Claim

Feed-forward 4D reconstructors, meaning networks that map a video to time-varying 3D primitives in one pass, inherit two shortcuts from narrow training data. The first is a motion shortcut: they memorize the kinematics of their simulator. The second is an appearance shortcut: they explain any photometric change as scene flow, the per-point 3D motion between frames. We claim both are dataset-diversity failures of the kind documented in image forensics. Both can be reduced by changing how training data is generated, while keeping the model fast.

The new ingredients are three. First, appearance-counterfactual training pairs, in which only the appearance changes and the supervision says the true scene flow is exactly zero. Second, a lightweight per-primitive appearance-residual output, which gives those changes somewhere legitimate to go. Third, a controlled study of motion-generator diversity versus sample volume at a fixed data budget.

The forensics precedent supplies the method and the hypothesis. Community Forensics trains fake-image detectors on 4,803 generators and finds that the number of distinct generators matters more than the number of images. IStego100K finds the same for randomized embedding algorithms in steganalysis. Nobody has run this experiment for 4D reconstruction. The models that motivate it state the gap themselves. DGS-LRM predicts deformable 3D Gaussians and dense scene flow from monocular video in about 0.6 s. It trains on 40,000 custom Kubric scenes, outputs color and deformation jointly, and reports that motion outside the Kubric distribution fails. 4RC factorizes a video into base geometry and relative displacement fields and trains on seven datasets. It still degrades on chaotic, high-frequency dynamics and does not model appearance at all.

How This Differs From the Obvious Combination

The obvious idea is to bolt Styl3R's design onto a 4D model. Styl3R is a feed-forward static reconstructor with a structure branch and an appearance branch. The structure branch is frozen while the appearance branch learns style, which keeps 2D brushstrokes from turning into 3D bumps. That freezing works only because the scene is static. In 4D the structure branch must keep predicting time-varying deformation, so freezing it would also freeze motion.

Our proposal does not rely on freezing. Every branch stays trainable. Separation is enforced by supervision that makes the shortcut costly: on appearance-counterfactual clips, any predicted flow on static surfaces is penalized directly.

The proposal is also distinct from Split4D. Split4D separates moving objects from the static background by attaching velocities and identity features to Gaussians, and it requires slow per-scene optimization. It separates objects from objects, not light from geometry, and it is not feed-forward.

Research Program

The data engine has two components.

Motion families. The engine composes scenes from a library of motion-generating programs, each defining one kinematic family. Families include rigid trajectories with different acceleration statistics, articulated skeletons, cloth and soft-body simulation, particle swarms, and erratic stochastic paths. The library is grown deliberately to include the chaotic and non-rigid motions that DGS-LRM and 4RC report failing on.

Appearance-change processes. These are rendered independently of motion: moving area lights and cast shadows, exposure and white-balance drift, flicker, animated textures on static geometry, and view-dependent specular highlights.

For a subset of scenes the engine renders matched clips that share the same camera path. One clip has motion with fixed lighting. The other has fixed geometry with changing lighting.

The model is a DGS-LRM-style transformer that predicts per-pixel Gaussians and scene flow. We add one extra output: a per-Gaussian, per-timestep color residual $\Delta c_t$, regularized toward zero. Training uses the usual photometric and flow supervision plus a leakage penalty on counterfactual clips:

$$\mathcal{L}_{\text{leak}} = \frac{1}{|S|}\sum_{i \in S}\lVert \hat{f}_i \rVert_1 .$$

Here $S$ is the set of primitives on surfaces that are static in the ground truth, and $\hat{f}_i$ is the predicted 3D flow of primitive $i$.

The program runs in three phases:

  1. Show the shortcut exists. Measure flow leakage of existing models on held-out appearance-only clips.
  2. Scale diversity at fixed volume. Hold the total number of clips fixed and vary the number of motion families, sweeping from a handful to the full library.
  3. Measure transfer. Test whether diversity and counterfactuals improve results on real monocular video.

Baselines

Naive reference point. A static feed-forward reconstructor, GS-LRM. It regresses Gaussians from two to four posed images in 0.23 s, and we apply it per frame. This sets the floor that any dynamic model must clear.

Strongest feed-forward competitor. DGS-LRM, retrained by us on the same total clip count, compute, and tuning budget as our model, using only its original Kubric-style data. It is the closest existing system: same output representation, same speed regime. The file reports approximately 14.89 PSNR for DGS-LRM on DyCheck and an ATE-3D of 0.21 on PointOdyssey (SpatialTracker: 0.22; chained RAFT-3D: 0.70). Beating it means three things together:

  • higher co-visibility-masked PSNR and PCK-T on the iPhone/DyCheck sequences;
  • equal or lower ATE-3D on PointOdyssey;
  • substantially lower flow leakage on our held-out appearance-change test.

A PSNR gain bought with worse leakage does not count.

Second feed-forward competitor. 4RC is the strongest diversity-scaled baseline. We evaluate it on geometry and motion metrics only, because it does not render appearance. For 4RC we found no numbers on Lacuna for the DyCheck or PointOdyssey metrics. The file reports only its 36% APD gain over V-DPM on Waymo and its best ATE on TUM-dynamics without a scalar.

Optimization reference. D3DGS-style per-scene optimization serves as the fidelity upper reference. DGS-LRM reports it takes 1 to 3 hours per scene.

Ablations. Each isolates one ingredient:

  • counterfactual pairs without the residual head;
  • the residual head without counterfactual pairs;
  • both, but with few motion families.

Why not just…?

Why not just use global photometric augmentation such as color jitter? This is the real threat, and we test it directly as a baseline with matched compute. Our hypothesis is that it falls short in a specific way. Global jitter changes every pixel equally, which the network can absorb as a per-frame offset. Moving shadows and sliding highlights are spatially structured and correlated with geometry, which is exactly what tempts a flow head. If jitter matches our leakage and DyCheck results, the counterfactual engine is unnecessary, and we will report that.

Why not just extend Styl3R's frozen structure branch to 4D? Freezing is what lets Styl3R keep geometry intact, but its authors restrict it to static scenes. A frozen deformation pathway cannot track new motion. We nonetheless run a variant that freezes the geometry and flow heads after the first training stage and trains only the appearance residual. If that variant matches ours on leakage but loses on PointOdyssey ATE-3D, that confirms the trade-off. If it loses on neither, freezing suffices.

Why not just add more real datasets, as 4RC does? Real dynamic datasets have no ground truth separating appearance change from motion. They also give no control over kinematic diversity, which is why 4RC's documented failures cluster on chaotic motion. As a further control, we train an otherwise-identical model on 4RC's real-plus-synthetic mixture at matched volume.

Proposed Experiments

The decisive test is the diversity-versus-volume sweep. If held-out-family ATE-3D and DyCheck PCK-T improve with the number of motion families at fixed clip count, the Community Forensics finding transfers to 4D. If they track only volume, the diversity hypothesis is wrong.

Because Monocular Dynamic View Synthesis: A Reality Check shows that multi-view leakage inflates scores, we evaluate only under its protocol. That means the iPhone sequences with co-visibility-masked PSNR, LPIPS, and PCK-T, where it reports a 4–5 dB PSNR drop and roughly a 30% PCK-T drop relative to leaky benchmarks. Because DGS-LRM itself trains with dual-view hints, we also report every model trained with and without them.

The counterfactual test set uses lighting processes and scenes held out from training. On it we report flow leakage on static surfaces and the accuracy of predicted appearance residuals. Finally, we confirm that inference stays under one second, so the per-scene-optimization advantage is preserved.