Turning a single phone video of a moving object into a full 360° animated 3D model (a "4D" asset) is a direct answer to the shortage of real 3D training data. Current systems get there by asking a 2D image generator to invent the views the camera never saw. Because each invented view is only weakly constrained, many systems generate frames one after another and let each frame borrow from the frames already produced. The video itself pins down the visible side at every moment. The hidden side, however, is anchored only by the model's own earlier guesses. Errors there can pile up over a long clip without the input video ever revealing them.
We propose a training-free procedure that makes this hidden drift visible and limits it. First, the sequential generator is run forward from a starting "keyframe" and backward from an ending keyframe. A keyframe is a timestep whose unseen views are generated fresh from that single frame, with no memory of other frames. The two runs pick up errors from opposite ends. Where they disagree on the hidden side, at least one of them has drifted. That disagreement tells us where to insert a new fresh keyframe, which memory entries to throw away, and how much to trust each run when fitting the final 3D model. The bet is that a forward-backward disagreement signal, built from nothing but the generator itself, tracks true hidden-view error closely enough to beat fixed anchoring schemes when the number of keyframes is held equal.
Core Claim
Video-to-4D generation sits between two failure modes. The first is generating each frame's novel views independently. This has no accumulated error, but the result jitters badly: per-frame Zero123 reaches a Fréchet Video Distance (FVD, a distance between distributions of real and generated videos, lower is better) of only 1571.60 in Consistent4D. The second is propagating information through memory. This is smooth, but it can drift and can poison its own memory, as Space-Time Memory Networks showed for 2D segmentation. We claim the best point between these is a schedule of fresh keyframes, and that the right schedule depends on the input. A slowly rotating toy needs almost no keyframes. A dancer turning around needs them exactly where the hidden side changes. Forward-backward disagreement is a cheap, label-free way to find those places. What is new is not memory, anchoring, or time reversal taken one at a time. It is using the disagreement between two oppositely anchored propagations as an estimate of error on the unobservable side, and using that estimate to schedule re-anchoring in 3D generation.
Relation to Prior Work
Three 4D-from-video systems motivate the problem.
Consistent4D optimizes a multi-scale space-time radiance field against a 3D-aware image diffusion prior, using a frame-interpolation consistency loss. That loss only regularizes adjacent frames, and the method takes about 2.5 hours per object.
Efficient4D generates a full grid of views by timestamps with a temporally smoothed multi-view generator, then fits 4D Gaussian splats with an optical-flow confidence weight. Its authors report that the smoothing makes independent parts move in lockstep. The confidence weight is also purely a 2D local check.
FB-4D is closest to us. It keeps a bank of attention keys and values from earlier generated frames and merges redundant or drifted entries into centroids. It reports the best FVD in the file, 724.26 on the Consistent4D benchmark. Its bank only ever looks backward, and nothing in it measures whether a merged entry has drifted on the hidden side. We reuse FB-4D's propagation as our forward and backward engine and add the drift estimate and scheduling around it.
Our design draws its mechanisms from three places. From 2D tracking we take the anchoring idea: FEELVOS pairs local matching with global matching to an uncorrupted first frame, and its ablation shows that local matching matters (J&F falls from 69.1% to 54.9% without it). We also take the memory gating idea from Mask Selection and Propagation and Detector-in-the-Loop Tracking. Both decide whether to trust a propagated prediction or a fresh independent detection. Our fresh keyframes play the role of that detector.
The time-reversal idea comes from BAgger. BAgger reverses drifted video rollouts and fine-tunes on them so the model learns to recover. We use reversal at inference instead, with no training, and in a setting where the quantity of interest (the hidden 3D side) is never observed.
VideoGPA constrains the design. It scores 3D consistency by reprojecting through a geometry foundation model and notes that this scales poorly with sequence length. That is why our disagreement is computed per timestep between two short sets of views, not by reconstructing the whole sequence.
A constraint we take seriously comes from the 2D tracking literature: global anchors fail under large appearance or topology change. This is why we re-anchor adaptively instead of tying everything to frame 0.
Research Program
The input is a monocular video $I_1,\dots,I_T$. We define a keyframe generator $G$: given a single frame, it produces a fixed set of novel views independently, as a SyncDreamer- or Zero123-style generator does. We define a propagator $P$: an FB-4D-style feature-bank generator that produces the views at time $t$ from the input frame $I_t$ and a memory seeded at a keyframe.
Between two consecutive keyframes $a<b$, we run $P$ forward from $a$ and backward from $b$, giving hidden-view image sets $V^{\rightarrow}_t$ and $V^{\leftarrow}_t$. The backward run applies the same generator to the time-reversed video, which is itself a valid video. Drift is scored as
$$D_t = d\big(V^{\rightarrow}_t, V^{\leftarrow}_t\big),$$
where $d$ is the perceptual distance between the two image sets (LPIPS averaged over views) plus a depth disagreement term from a monocular depth estimate. Each view set is first checked against the observed frame $I_t$, so that a run that is wrong on the visible side is caught directly.
Keyframes are placed greedily. We start with the endpoints. While the keyframe budget $K$ allows, we insert a fresh keyframe at the timestep with the largest $D_t$ above a threshold and rerun only the two affected segments.
The final 4D Gaussian splat model is fit to a fused target. At each $t$, each run is weighted by its distance from its own seed keyframe and by $D_t$. Specifically, the forward weight is proportional to $\exp(-\lambda (t-a)) / (1 + D_t)$, and the backward weight is defined symmetrically. Here $\lambda$ is a single drift-rate constant fitted on held-out synthetic sequences. Supervision on hidden views with high $D_t$ is down-weighted, replacing Efficient4D's flow-based confidence. Memory entries written at timesteps with high $D_t$ are left out of the bank, which addresses poisoning directly.
Inference costs roughly two propagation passes plus $K$ independent generations. We report wall-clock time against FB-4D and Efficient4D (Efficient4D reports 10 minutes on an A6000).
Baselines
The naive reference point is per-frame independent generation followed by the same 4D Gaussian fit. This is the "all keyframes, no memory" end of the trade-off, and it shows what jitter costs. Its reported reference is FVD 1571.60 for per-frame Zero123 in Consistent4D. We rerun it in our pipeline so that the reconstruction stage is identical across methods.
The strongest existing method is FB-4D. It is the "one keyframe, all memory" end, includes its own drift-mitigating merge, and reports FVD 724.26 on the Consistent4D benchmark (against STAG4D at 992.21). Since our method wraps FB-4D's propagator, the comparison is unusually clean. Both use the same diffusion weights, the same view set, the same 4D Gaussian fitting code, and the same number of fitting iterations. Our extra generator calls are reported, not hidden. On the standard short benchmark, beating FB-4D means a lower FVD than our own rerun of FB-4D, plus LPIPS and CLIP that are at least as good. On long sequences, it means a clearly lower slope of hidden-view error against frame index.
Efficient4D is run as the joint, non-autoregressive alternative, with a 0.92 CLIP score reported. Consistent4D (FVD 992.61) serves as the optimization-based reference.
Four ablations isolate the key ingredient, and each matches $K$:
- Forward-only propagation with keyframes at the same positions our method chose. This isolates the bidirectional fusion.
- Bidirectional propagation with evenly spaced keyframes. This isolates adaptive placement.
- Keyframes placed at the largest frame-to-frame change in the input video. This is a cheap visible-side heuristic.
- Keyframes placed using ground-truth hidden-view error. This is an oracle upper bound and is available only on synthetic data.
Why not just…?
Why not just anchor everything to frame 0, as FEELVOS does? Global anchoring assumes the reference stays visually relevant. The research file notes that this assumption breaks under large non-rigid or topological change, and a turning or deforming object is exactly that case. A frame-0 anchor would also miss new hidden-side content that appears halfway through the clip. We include this as an explicit variant: FB-4D with its bank permanently pinned to frame 0. It may match us on slowly moving objects, and we will report that honestly. We expect it to lose on sequences where the back of the object changes.
Why not just place keyframes every N frames? This is the most serious threat. If drift grows at a roughly constant rate on every input, evenly spaced keyframes are nearly optimal, and our disagreement signal adds cost for nothing. Ablation 2 settles this directly at matched $K$. We will also report how much drift rate varies across sequences, since our method only helps when that variation is large. If even spacing comes within noise of our method, we will conclude that the useful contribution is the bidirectional fusion alone, not the scheduling.
Why not just generate all frames jointly so nothing accumulates? Efficient4D's grid generation avoids sequential drift. The cost is temporal smoothing that its authors document as motion entanglement. Joint generation is also limited by how many timestamps fit in one pass. On the standard short clips it is a fair competitor and may win on speed. Our long-horizon test is designed to show what happens past that limit. We will also measure entanglement, as the correlation of motion between parts that should move independently.
Why not just use a bigger memory bank? STM's poisoning result and FB-4D's need for merging both show that a larger bank stores more corrupted entries. A larger bank adds no information about the hidden side that the generator did not already invent. We run FB-4D with double the bank size as a control.
Proposed Experiments
The first experiment is a drift diagnosis, because the whole proposal depends on it. We render animated Objaverse assets, the repository used by VFusion3D, as long front-view videos of 120 to 300 frames, with ground-truth renders from a ring of hidden views. Across many such sequences, we measure how FB-4D's hidden-view error grows with frame index, and whether $D_t$ correlates with true hidden-view error. A Spearman rank correlation is reported per sequence, and per frame relative to the visible-side error, which is the obvious and cheaper competitor signal.
Two outcomes would kill the method. If hidden-view error does not grow with horizon, the motivation fails. If $D_t$ correlates no better than visible-side error, the drift estimate fails. In either case we report the negative result instead of the method.
If the diagnosis holds, the second experiment is the matched-$K$ comparison on these long sequences. The metrics are hidden-view PSNR and LPIPS against ground truth, error slope against frame index, and the gap to the oracle keyframe placement.
The third experiment is the standard Consistent4D benchmark, reporting FVD, LPIPS and CLIP against the reported numbers above and against our reruns. This checks that the method does not regress on short clips, where we expect only small gains.
STAG4D and HD-EPIC-style in-the-wild clips have no reported numbers on Lacuna, so they receive qualitative evaluation and a user preference study, following the 75% preference protocol used in Consistent4D.
References
- Consistent4D: Consistent 360° Dynamic Object Generation from Monocular Video
- Video Object Segmentation Using Space-Time Memory Networks
- Efficient4D: Fast Dynamic 3D Object Generation from a Single-view Video
- FB-4D: Spatial-Temporal Coherent Dynamic 3D Content Generation with Feature Banks
- FEELVOS: Fast End-To-End Embedding Learning for Video Object Segmentation
- Mask Selection and Propagation for Unsupervised Video Object Segmentation
- Detector-in-the-Loop Tracking: Active Memory Rectification for Stable Glottic Opening Localization
- BAgger: Backwards Aggregation for Mitigating Drift in Autoregressive Video Diffusion Models
- Distilling Geometry Priors for 3D-Consistent Video Generation
- VFusion3D: Learning Scalable 3D Generative Models from Video Diffusion Models