Take a casual phone video in which both the camera and the people in front of it move. To turn that video into 3D, current pipelines run a pretrained single-image depth network on every frame, then solve for the camera path and a few correction parameters per frame so that the depth maps agree with one another. Two different things can make depth maps from two frames disagree. The depth network may simply be wrong in some region, for example by underestimating the distance to a car. Or the car may actually have moved. Methods that assume a static world, such as FrozenRecon, treat every disagreement as depth error and bend a local correction field until it disappears. Methods built for moving objects treat disagreement as motion and either discard those pixels or model them as deforming. Neither family asks which explanation is correct, even though the two leave different fingerprints in 3D.
The fingerprint is simple. If the depth network is wrong by a factor in some region, the object reconstructed there is inflated or shrunk in 3D. If the object moved rigidly, or nearly rigidly over a small patch, its 3D size stays the same. We propose a test-time optimization, with no training, that keeps a frozen depth model and adds two competing explanations for each tracked patch: a local depth-scale correction and a rigid patch motion. The correction is allowed to explain only changes in size, and the motion only displacement that preserves size. The output is a camera trajectory, full-scene per-frame point maps covering both the static background and the moving foreground at one consistent scale, and a per-patch answer to "moving or mis-scaled?". Because an object's size must stay constant over time, the same constraint also corrects the depth network's errors on moving objects, which masking-based pipelines cannot do.
Core Claim
In dynamic video, the frozen single-image depth model is the one component that does not assume a static world, because each prediction depends on only one frame. Its errors and independent scene motion are confounded only if the optimization is free to trade one against the other. If size changes are assigned to depth correction and size-preserving displacements to motion, then (a) camera poses estimated from patches labeled static become robust to moving objects, (b) moving objects receive temporally consistent metric depth, and (c) static regions where the depth model is badly wrong are no longer mistaken for motion. The approach is expected to fail in two cases, and we treat both as findings to measure rather than hide. The first is a small, distant object moving along the viewing ray, where size change falls below the noise. The second is a scene in which a moving foreground covers most of the image, leaving too few static patches to anchor the camera.
Position Relative to Prior Work
Three static-world methods supply the machinery we reuse. FrozenRecon freezes LeReS, optimizes about 30 parameters per frame (camera, focal length, scale, shift), and fits local scale and shift fields by locally weighted regression around anchor points. NoPe-NeRF refines per-frame scale and shift of DPT depth jointly with poses using a Chamfer loss between consecutive frames. Look Gauss, No Pose optimizes poses inside Gaussian splatting. The research file notes that all three assume rigidity. FrozenRecon's local field is the specific mechanism our proposal targets: it is flexible enough to absorb a moving object as "depth error", which distorts that region and corrupts the global scale. We keep its sparse parameterization and constrain what the local field may explain.
Distractor-robust methods motivate the proposal and also limit it. RobustNeRF trims the highest-residual half of pixels. NeRF On-the-go learns an uncertainty map from DINOv2 features. RobustSplat and DeGauss separate transient from static content in Gaussian splatting. All of them need precomputed poses, and all of them discard the moving content or model it apart from the background. None can tell a moving object from a static region where the depth is wrong, since both produce high residuals.
The closest work spans all three directions. PAGE-4D fine-tunes the middle layers of VGGT to predict a soft dynamics mask. It suppresses dynamic tokens for pose estimation and keeps them for geometry, in a single feed-forward pass. Our method differs in three ways: it is training-free optimization rather than a learned network, it separates motion from depth error with an explicit geometric test rather than learned attention, and it works with any frozen depth model. PAGE-4D also reports that its pose estimate degrades when a dominant foreground starves the static tokens, and we expect a version of the same failure. DynSUP solves camera and per-object rigid transforms from Depth Anything V2 depth and SAM-2 segments, but only for two frames, and it takes segment boundaries from SAM-2. We instead use small patches with local rigidity, apply the method to whole videos, and never ask whether a segment is right. Free-DyGS jointly optimizes poses and deformation but needs stereo depth. PAD3R discards the background and depends on a generative 3D proxy.
Research Program
The inputs are a monocular video, per-frame affine-invariant depth $D_i$ from frozen Depth Anything V2 (Depth Anything V2), and dense 2D correspondences from an off-the-shelf flow model. Each frame is divided into small patches of tracked points. For frame $i$ we optimize a camera pose $G_i$, focal length, global scale and shift, and a smooth low-dimensional local scale field $s_i(u)$, all as in FrozenRecon. For each patch $P$ and frame pair $(i,j)$ we add a rigid motion $T_P^{ij}$.
The lifted point is $X_i(u) = G_i\,\pi^{-1}(u, s_i(u)D_i(u))$, where $\pi^{-1}$ back-projects pixel $u$ at the given depth. The objective is
$$ \min_{G, s, T}\ \sum_{P,(i,j)} \sum_{u\in P} \rho\big(\lVert T_P^{ij} X_i(u) - X_j(u')\rVert\big) + \lambda_m \sum_{P,(i,j)} \lVert \log T_P^{ij}\rVert_1 + \lambda_s\, R(s), $$
where $u'$ is the correspondence of $u$ in frame $j$, $\rho$ is a robust penalty, the $L_1$ term on the motion's Lie-algebra coordinates pushes most patches to remain exactly static, and $R$ penalizes roughness in the scale field.
On its own, this objective is ambiguous: $s$ and $T$ can each absorb the other's residual. The key ingredient breaks the ambiguity. A rigid $T$ preserves distances, so the ratio of intra-patch distances
$$ r_P^{ij}(a,b) = \frac{\lVert X_j(a') - X_j(b')\rVert}{\lVert X_i(a) - X_i(b)\rVert}, \quad a,b \in P, $$
depends only on the scale field. We therefore enforce $r_P^{ij}\approx 1$ as a separate loss on $s$, and stop gradients from the residual term into $s$ for patches with $T\neq I$. This makes size constancy the only signal that drives depth correction on moving content. A patch counts as dynamic if its fitted motion magnitude exceeds a threshold set from static-scene noise. A per-patch identifiability score, the expected size change from the fitted motion along the ray relative to correspondence noise, flags cases the method cannot resolve.
Optimization proceeds coarse to fine. Cameras are initialized from a robust closed-form rigid alignment of consecutive lifted depth maps, followed by sliding-window refinement. The core deliverable is geometry. As a secondary test of whether it helps rendering, the resulting cameras and point maps initialize an existing dynamic Gaussian representation on DyCheck.
Baselines
The naive reference is Depth Anything V2 run frame by frame, with each frame's scale and shift fitted to ground truth by median alignment for depth metrics, and with cameras from a static robust alignment for trajectory metrics. It shows how far raw single-image depth gets when nothing reasons across frames.
The static pose-free baseline is FrozenRecon with its local scale field. To isolate the effect of method rather than depth model, we rerun it with Depth Anything V2 in place of LeReS, using the same flow, iteration budget, and sliding window as our method. It is the strongest test of whether moving objects actually hurt the local-field approach. Our method must lower its trajectory error on dynamic sequences of TUM and Sintel, while matching it within noise on static ScanNet, where it should lose nothing.
The robust-loss baseline is FrozenRecon with a trimmed geometric residual in the style of RobustNeRF: at each iteration, keep the lowest-residual 50 percent of points. It answers whether a generic outlier estimator is enough.
The mask baseline is FrozenRecon run on background pixels only, using PAGE-4D's soft dynamics mask thresholded at a level tuned on a held-out split. Foreground pixels are then placed using each frame's fitted scale. This is the pipeline a skeptic would assemble from existing parts.
The strongest existing method is PAGE-4D itself. The file reports, for PAGE-4D on Sintel, video-depth Abs Rel 0.212, $\delta<1.25$ of 0.763, and ATE 0.143. On DyCheck it reports mean point-map accuracy error 0.403. On TUM it reports only a 21 percent translation-error reduction relative to VGGT, with no absolute value found on Lacuna. We will run its released model under our own evaluation protocol rather than copying numbers. Beating it means lower Sintel ATE and Abs Rel and lower DyCheck accuracy error under that shared protocol. We also compare compute honestly: PAGE-4D runs at 43.2 FPS, while our method is per-scene optimization measured in minutes.
The key ablation removes the size-constancy loss and the gradient stopping while keeping everything else, so that $s$ and $T$ compete freely. If this variant performs as well, the central claim is false.
Why not just…?
Why not just mask moving pixels with a segmenter or PAGE-4D and run static pose-free reconstruction on the rest? This is the most serious threat, and the mask baseline is built to test it. Masking addresses camera robustness, but it leaves two problems. First, foreground depth inherits the depth model's per-frame errors uncorrected, because the mask carries no constraint on the object's own geometry. Size constancy supplies exactly that constraint. Second, a static region where the depth is badly wrong is not flagged by a motion mask, so a local field must still absorb it. In addition, upstream mask errors propagate into the result, as DynSUP notes, and Split4D reports that tracker-dependent decomposition reaches 0.592 mIoU against 0.893 for its native clustering. If masking matches us on ATE, our contribution narrows to foreground depth accuracy. We will measure that separately as temporal Abs Rel restricted to dynamic pixels on Sintel, and as the variance of object 3D extent over time.
Why not just use a trimmed or robust loss? Trimming ranks points by residual size, and both motion and depth error produce large residuals. It therefore discards badly estimated static regions along with movers, and it throws away the foreground entirely. The robust-loss baseline and the synthetic diagnostic below measure this directly.
Why not just trust PAGE-4D's feed-forward output? It may already be good enough on Sintel-like data. However, it does no test-time refinement over long sequences, its point maps carry no explicit cross-frame size constraint, and its mask was learned from a particular training distribution. A training-free method that wins on DyCheck's real monocular phone videos would show the geometric test transfers where learned masks may not. If we lose on every benchmark, the honest conclusion is that learned attention already resolves the ambiguity.
Why not just add time as an input to a Gaussian or radiance field and fit everything end to end? Monocular Dynamic View Synthesis: A Reality Check shows that on true monocular iPhone captures, existing dynamic models lose 4 to 5 dB in PSNR and about 30 percent in PCK-T, the keypoint-transfer accuracy metric. This suggests that photometric fitting alone does not recover correct geometry, before unknown poses are even considered.
Decisive Experiments
The first experiment is a controlled diagnostic on Sintel and on synthetic perturbations of ScanNet, a static dataset. We inject known local scale errors into static regions, insert rigidly moving objects with known motion, and in some cases do both. We then measure how accurately each method labels patches as moving or mis-scaled, and how classification error depends on the identifiability score. The prediction is that accuracy is high above the score threshold and near chance below it, along the viewing ray. The free-competition ablation should confuse the two causes everywhere.
The second experiment is the main benchmark suite. On Sintel and dynamic TUM sequences we report ATE and video-depth Abs Rel and $\delta<1.25$. On DyCheck we report point-map accuracy, evaluated without camera teleportation as the Reality Check paper recommends. On ScanNet we confirm no loss in static scenes.
The third experiment is a stress test of dominant foregrounds. We bin sequences by the fraction of the image that is moving and plot trajectory error for our method and for PAGE-4D, to see whether test-time optimization fails more gracefully than token starvation, or just as badly.
If the size-constancy ablation matches the full method, or if the mask baseline matches it on both camera error and foreground-depth error, we will report that the separation of motion from depth error is not the bottleneck.
References
- FrozenRecon: Pose-free 3D Scene Reconstruction with Frozen Depth Models
- NoPe-NeRF: Optimising Neural Radiance Field with No Pose Prior
- Look Gauss, No Pose: Novel View Synthesis using Gaussian Splatting without Accurate Pose Initialization
- RobustNeRF: Ignoring Distractors with Robust Losses
- NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- DeGauss: Dynamic-Static Decomposition with Gaussian Splatting for Distractor-free 3D Reconstruction
- PAGE-4D: Disentangled Pose and Geometry Estimation for VGGT-4D Perception
- DynSUP: Dynamic Gaussian Splatting from An Unposed Image Pair
- Free-DyGS: Camera-Pose-Free Scene Reconstruction based on Gaussian Splatting for Dynamic Surgical Videos
- PAD3R: Pose-Aware Dynamic 3D Reconstruction from Casual Videos
- Depth Anything V2
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- Monocular Dynamic View Synthesis: A Reality Check