Time as a Witness: Checking Generated Novel Views of Deforming Scenes Against Real Frames

Suppose you want a 3D model of a person dancing or a dog shaking itself dry, and all you have is one ordinary video. One popular shortcut is to ask a video generator to "re-film" the clip from new camera angles and then treat those invented views as if a multi-camera rig had recorded them. The generators that do this well are mostly trained on synthetic scenes from game engines. On real footage they produce views that look plausible but are subtly wrong, and nothing in the pipeline can tell which parts are wrong. This proposal points to a signal that is already sitting in the monocular video. When a scene moves and bends, surfaces hidden at one moment are exposed at another. With a good model of how each surface point moves over time, a model of non-rigid motion, every pixel the generator invents for a new camera at time $t$ can often be checked against a real pixel the original camera saw at some other time $t'$.

We propose to use these "temporal witnesses" for two purposes. The first is to adapt a synthetic-trained, camera-controlled video generator to real footage without any real multi-view data. The second is to decide which generated pixels are reliable enough to supervise a downstream reconstruction of the dynamic scene. The claim is that non-rigid motion, which is usually treated as the obstacle in dynamic reconstruction, is the source of the multi-view evidence needed to close the synthetic-to-real gap.

Core Claim

A monocular video of a deforming scene contains multi-view evidence spread across time. If we lift the video into 3D point trajectories, using per-frame depth estimates plus long-range 2D point tracking, then any point rendered into a virtual camera at time $t$ can be followed along its trajectory to a time $t'$ at which the real camera observed it. The generated color can then be compared to the real one. This comparison gives a supervision signal on real data for exactly the content a generator hallucinates: disoccluded regions of a new view. Same-time warping cannot check that content, and synthetic training data cannot calibrate it for real appearance. Our prediction is that fine-tuning on this signal will improve held-out real-camera fidelity. We also predict that masking pseudo-views by witness disagreement will improve dynamic reconstruction beyond what robust fitting on unfiltered pseudo-views achieves.

Where This Sits

The method we build on is Beyond Inpainting: Unleash 3D Understanding for Precise Camera-Controlled Video Generation. That work, DepthDirector, re-renders a video along a new camera path. It conditions on warped depth rather than warped RGB, because warped RGB leads to what the authors call the "Inpainting Trap": the generator copies warping distortions into its output as though they were ground truth. DepthDirector is trained with a small adapter on MultiCam-WarpData, 8,000 synthetic Unreal Engine 5 videos across 1,000 dynamic scenes.

That paper motivates our problem in two ways. It shows that synthetic multi-camera data is enough to teach camera control. But its training signal comes entirely from simulation, so nothing checks whether its hallucinated content is correct on real video. We keep its depth-conditioned design and add a real-data correction stage it does not have.

Two nearby lines of work supply the underlying principle. MVP-LAM: Learning Action-Centric Latent Action via Cross-Viewpoint Reconstruction and Multi-view Consistent Latent Action Learning for World Modeling and Control both learn action representations by requiring that what is inferred from one viewpoint reconstruct another. They show that cross-view reconstruction removes viewpoint bias, but they assume the other views exist. We build the "other view" from the same camera at a different time. GraftSR: Grafting Authentic Textures for Real-World Image Super-Resolution via Identical-Instance Guidance avoids texture hallucination by drawing detail from real observations of the same instance. Our witness loss applies the same idea, grounding generation in authentic pixels, across time in a deforming scene rather than across images of a static instance.

Research Program

The pipeline has three stages.

In the first stage, we lift a real monocular video into a set of 3D point trajectories $x_p(t)$. Here $p$ indexes a tracked surface point and $t$ indexes the frame. We build these from monocular depth and dense long-range tracks, and keep only tracks that are forward-backward consistent. Each point also gets a visibility label per frame from depth ordering and track occlusion flags. These trajectories are our non-rigid motion model. We deliberately use an explicit, inspectable form rather than a learned deformation network, so that tracking errors can be measured separately from generator errors.

In the second stage, we render depth sequences from these trajectories into virtual cameras $c$. DepthDirector then generates the corresponding videos $V^c$. The witness loss for a point that is visible in camera $c$ at time $t$ but was visible to the real camera only at some other time $t'$ is

$$ \mathcal{L}_{\text{wit}} = \sum_{c,t,p} w_p \, d\big(V^c_t(\pi_c(x_p(t))),\; I_{t'}(\pi_0(x_p(t')))\big). $$

In this equation, $\pi_c$ and $\pi_0$ project a 3D point into the virtual and real cameras, $I_{t'}$ is the real frame, $w_p$ is the track's confidence, and $d$ is a distance in a pretrained feature space combined with a per-frame affine color correction. That correction gives tolerance for shading that changes as the surface bends. We choose $t'$ as the frame where the point is visible and its surface faces the real camera most directly.

The loss is used in two ways. We add it to the adapter-tuning objective on unlabeled real videos, which adapts the generator from simulation to real footage. We also threshold the per-pixel residual into a reliability mask for downstream use.

In the third stage, we fit a deforming 4D scene representation to the real video together with the masked pseudo-views. We then measure reconstruction quality from held-out real cameras.

Our analysis will separate the three places error can come from: tracking, generation, and the reconstruction fit. We will do this by stratifying results by track confidence and by the time gap $|t - t'|$.

Baselines

The naive reference point fits the same 4D representation to the monocular video alone, with no generated views. It tells us whether pseudo-views help at all on real deforming scenes, and it establishes the depth-and-tracking prior that every other variant shares.

The strongest existing competitor is off-the-shelf DepthDirector, used to generate pseudo-views that are then passed to the same 4D fit. It is the strongest competitor because it already avoids the main failure of RGB warping and is the only camera-controlled re-renderer in the file with reported numbers. On its own evaluation, Beyond Inpainting reports a VBench subject-consistency score of 95.29 and a human-centric reference similarity of 0.6887. We will run it with its released weights and identical trajectories and depth inputs. Its adapted counterpart will start from the same weights, adapter rank, and step budget.

Beating DepthDirector means two things. The first is a statistically reliable gain in held-out-camera fidelity (PSNR and LPIPS) on real multi-view dynamic footage. The second is not falling below 95.29 subject consistency on VBench, so that the gains do not come at the cost of temporal coherence.

A third baseline gives DepthDirector's unfiltered pseudo-views to the 4D fit with a robust loss, such as a heavy-tailed photometric penalty. This tests whether our explicit mask adds anything beyond generic outlier rejection.

Two ablations isolate the key ingredient. The first restricts witnesses to $t' = t$, so only same-time co-visible pixels are checked, which removes the temporal part. The second replaces the non-rigid trajectories with a single rigid motion per frame, which is effectively a static-scene assumption. If both ablations match the full method, the non-rigid temporal witness is not doing the work.

Why not just…?

Why not just scale or randomize the synthetic data? Domain randomization in Unreal Engine 5 is the default remedy, and DepthDirector's efficiency, using only 5.8% of the data needed by implicit camera-control baselines, suggests more synthetic data is affordable. But randomization widens the synthetic distribution without ever showing the model real deformable materials, cloth, skin, or real sensor noise. We will train a variant on doubled, randomized synthetic data under the same compute budget. If it matches the witness-adapted model on real held-out cameras, our claim fails.

Why not just fit a dynamic reconstruction to the monocular video directly? This is the naive baseline above. The limitation it cannot escape is that surfaces never seen from the input camera receive no gradient. The witness method only helps where some other frame saw the surface. We expect gains concentrated in regions exposed by motion and no gain in regions that are never observed, and we will report results split this way.

Why not just use generated views with robust fitting? This is a real threat. Robust losses already down-weight inconsistent pixels, and a 4D fit across many pseudo-views may reject hallucinations by averaging over them. The difference is that robust fitting only removes content that disagrees with other generated views. If the generator is consistently wrong about real appearance, as a synthetic-trained model may be, the error survives. The third baseline settles this directly. If it matches our method, we would keep only the adaptation use of the witness loss and drop the filtering claim.

Why not just fine-tune the generator on real video by predicting held-out frames? Frame prediction from the same camera never exercises camera control, so it cannot correct view-dependent errors. We will include it as an additional adaptation baseline with an equal step budget.

Proposed Experiments and Decisive Checks

Adaptation will use unlabeled real monocular videos of deforming subjects. Synthetic sanity checks will use MultiCam-WarpData, where true novel views exist. There, the witness residual should correlate with true error against the ground-truth renders. If it does not, the signal is not worth using on real data.

Real evaluation requires synchronized multi-camera captures of dynamic, non-rigid scenes, used strictly for testing. No real multi-view dynamic benchmark or reported result was found on Lacuna, so we will either use a public capture set or record a small rig evaluation set. We will report held-out-camera PSNR and LPIPS, VBench subject consistency, and reference similarity, and we will compare numbers only within our own runs.

The decisive checks are three. First, the full method should beat both the off-the-shelf and robust-fit DepthDirector baselines on held-out real cameras. Second, the gain should shrink under the $t' = t$ ablation and under the rigid-motion ablation. Third, the gain should grow with the fraction of each scene exposed by motion, since that dependence is the method's defining prediction.

A clear negative outcome would also be informative. If tracking errors on real non-rigid footage make the witness residual no more predictive of true error than the robust-fit residual, that would show that point tracking, not generation, is the bottleneck for this approach.