Motion-Aware Local Scale Alignment for Feed-Forward 4D Reconstruction

Single-image depth networks get the shape of a scene roughly right but not its true size, and their errors are not uniform. Planes come out curved, the image periphery warps, and some regions are too far while others are too near. A well-known fix is to take a few dozen trusted metric depth points, for example from a SLAM tracker or a sparse LiDAR, and fit a separate scale and offset around each pixel instead of one for the whole image. That fix assumes nothing in the scene moves. Fast feed-forward 4D models, which reconstruct moving scenes from video in one network pass, have the opposite limitation. They handle motion, but they ground metric size with a single global number per sequence or view. This proposal builds the missing piece: a feed-forward alignment layer that turns sparse metric anchors into a per-pixel, per-frame scale-and-offset field. The layer learns which anchors to trust and how far each anchor's influence should spread. Because it knows what is moving, it does not let a mis-triangulated point on a walking person bend the wall behind them.

The question matters for a second reason. When base depth is warped, a 4D model's motion estimates have to absorb that warp. The model then reports "phantom motion" on static background that is really a depth error. Correcting the depth warp before motion is estimated should show up as static points that stop moving. That is a clean, measurable consequence, and it separates this proposal from simply training a better depth network.

Core Claim

We claim that local scale alignment helps in dynamic scenes only if its weighting respects motion. We test this claim with a specific design: a closed-form weighted least-squares solve embedded in the network, whose per-pixel anchor weights and per-anchor reliability are predicted from image and motion features. Concretely, for pixel $p$ at frame $\tau$, with backbone depth $d$ and anchors $j$ carrying metric depth $z_j$ at backbone depth $d_j$, we compute

$$ (s_p, t_p) = \arg\min_{s,t} \sum_j \rho_j\, w_{pj}\,(s\, d_j + t - z_j)^2 + \lambda\big[(s - \bar s)^2 + t^2\big]. $$

Here $w_{pj}$ is a learned affinity between pixel $p$ and anchor $j$, built from image distance, feature similarity, and a motion-boundary term. $\rho_j \in [0,1]$ is a learned anchor reliability. $\bar s$ is the backbone's global metric scale, which acts as a prior, and $\lambda$ is a learned regularization strength. Each pixel's problem is a 2×2 linear system, so the layer is non-iterative and differentiable.

The corrected depth $s_p d_p + t_p$ then replaces the backbone depth before the motion head runs. The design keeps the inductive bias that made locally weighted regression work and replaces its fixed Gaussian kernel with one that does not cross depth discontinuities or moving-object boundaries.

Relation to Prior Work

The motivating evidence comes from Toward 3D scene reconstruction from locally scale-aligned monocular video depth. That work shows monocular depth errors are spatially heteroscedastic, meaning their size varies across the image. It corrects them with locally weighted linear regression over 25–100 SLAM or SfM anchors and reports relative-error drops of 54% on ScanNet and 48% on KITTI. Two further papers support the same picture. Monocular Visual-Inertial Depth Estimation finds that globally aligned depth bends planes, and FrozenRecon observes warping near frame borders. The regression method is run per frame at test time with a fixed distance kernel, and it assumes a static world. We reuse its solve, learn its kernel, and remove the static assumption.

The application regime comes from feed-forward 4D models. Any4D factors a video into per-frame depth and intrinsics plus global poses and scene flow. It handles 50 frames in 0.5 s and grounds size with a global metric scale head. 4RC encodes a video once and queries base geometry and motion displacements, all in normalized space. BTimer regresses pixel-aligned Gaussians at a chosen timestamp and has no metric grounding. AMB3R is static-only and sets metric size from one median log-depth per view. None of these can consume sparse metric anchors as a spatially varying correction. We add exactly that, on top of a frozen backbone.

The closest neighbor is Splat-SAP. It predicts a global scale from known stereo baselines plus a GRU-refined pixel-wise offset map for human-centered scenes. It needs calibrated multi-view rigs and does not condition on anchors or reason about anchor reliability under motion.

Two lines of work constrain the design. Distractor-rejection methods such as DGGS and RobustSplat delete moving content. We keep moving objects and must align them too. Split4D reports that 2D video segmentation fails under fast motion and occlusion: its mIoU is 0.893 on Multi-Human against 0.592 for the segmentation-lifting SA4D. This is why we do not make a hard segmentation mask the only defense against bad anchors.

Metric3D v2 supplies a strong single-image metric prior, with 0.044 AbsRel on KITTI. However, it relies on known focal length and normalizes the camera globally, so it is a candidate input rather than a solution.

Research Program

We build the alignment layer as a lightweight head on top of a frozen 4D backbone. Any4D is the primary backbone; we use 4RC as a second backbone to show the layer is backbone-agnostic. The head receives the backbone's depth, its motion features, image features, and a sparse anchor set. From these it predicts the affinity $w_{pj}$ as a softmax over anchors of a learned score. The score combines image distance with feature similarity and a penalty for crossing predicted motion or depth boundaries. The head also predicts the reliability $\rho_j$ from the anchor's local residual and how consistent the anchor is with the backbone's own motion estimate. The corrected depth is passed back into the backbone's motion head, which we fine-tune lightly, so that motion is inferred from unwarped geometry.

The training signal comes from datasets with dense ground-truth depth and motion: Dynamic Replica and PointOdyssey, plus Waymo with LiDAR. Anchors are simulated to reproduce the failure we care about. Static anchors are drawn from ground truth with SLAM-like noise. Anchors on moving objects are generated the way a static-world triangulator would produce them, by triangulating the object's track as if it were stationary, which gives systematically biased depth. On Waymo we also use real sparse LiDAR beams, in the spirit of LiDARTouch (Zablocki). Those anchors are metrically correct even on movers, which tests whether $\rho$ learns source-dependent trust rather than "moving means bad."

Moving objects that end up with no trusted anchors get their correction from the prior $\bar s$ and from anchors at their ground contact, through the learned affinity. We report their error separately so this weakness is not hidden. The loss is metric depth error plus scene-flow error, with an extra term that penalizes nonzero predicted motion on ground-truth static points.

Baselines

The naive reference fits one global scale and shift per frame to the same anchors by least squares and applies it to the backbone depth. This is what most pipelines do, and it shows how much local alignment buys at all.

The strongest existing feed-forward method is Any4D used as published, with its own global metric head. Its reported result on file is APD 93.44 on Dynamic Replica against 62.34 for SpatialTrackerV2 (Any4D). We found no reported Any4D depth AbsRel on Lacuna, so we re-measure depth ourselves on identical frames. For Waymo tracking, 4RC reports a 36% APD gain over V-DPM (4RC), and we run it as a second strong reference.

The strongest alignment baseline is fixed-kernel locally weighted regression from Toward 3D scene reconstruction from locally scale-aligned monocular video depth. It is applied to the same backbone depth and the same anchors, with its kernel bandwidth tuned on the same validation split we use. Its reported gains are relative: a 54% Rel drop on ScanNet and 48% on KITTI over global alignment. We first reproduce that gap on static ScanNet to confirm our implementation before testing it on dynamic data.

We also run a direct-regression baseline. It is a depth-completion-style refinement network of matched parameter count that takes backbone depth and the rasterized anchors and regresses corrected depth, trained on the same data for the same number of steps.

All methods share the frozen backbone, the anchor sets, the training data, and the GPU budget. Learned baselines receive the same hyperparameter-search budget as our head.

Two ablations isolate the key ingredient. The first removes the motion-boundary term and the reliability gate, which leaves a learned but motion-blind kernel. The second keeps them but bypasses the re-injection of corrected depth into the motion head.

Beating the field means three things on dynamic regions of Dynamic Replica and Waymo. Background AbsRel must be lower than the best of the fixed-kernel and masked baselines, with the gap growing as the fraction of contaminated anchors grows. Phantom motion on static points must drop relative to Any4D. APD must not fall below the published 93.44.

Why not just…?

Why not just fit one global scale and shift to the anchors? The local-alignment literature already answers this for static scenes: global fits leave curved planes and double walls, and local fitting cut relative error roughly in half on ScanNet and KITTI (Toward 3D scene reconstruction from locally scale-aligned monocular video depth, Monocular Visual-Inertial Depth Estimation). What remains open is whether this gap survives in dynamic video with a modern backbone whose warps may be smaller. Our naive baseline measures exactly that. If the gap closes, the proposal's premise fails, and we would report it.

Why not just mask moving objects with a segmenter and run standard local regression on the remaining anchors? This is the real threat, and the first decisive comparison. Masking addresses contamination but has two known costs. Segmentation drifts under fast motion and occlusion, as Split4D documents. Masking also discards correct anchors on movers, such as LiDAR returns, and leaves moving objects with no local correction at all. A fixed kernel further blurs scale across object boundaries even when every anchor is clean. We run this baseline with an oracle mask from ground-truth motion as well as a predicted mask. If the oracle-masked variant matches our method on both background and moving-object error, then the learned reliability adds nothing, and we would reframe the contribution as the boundary-respecting kernel alone.

Why not just train a network to regress corrected depth directly from backbone depth and anchors? A dense depth head can in principle represent any correction, so the solve is an inductive bias rather than added capacity. Our bet is that the closed-form solve generalizes better when anchor count and layout change: 25 versus 100 anchors, SLAM versus LiDAR patterns. A regression network tends to overfit the anchor distribution it was trained on. We test this by training both at one anchor density and evaluating across densities and sources. If direct regression matches us across that range, the structured layer is unnecessary.

Why not just feed Metric3D v2 depth into the 4D model? Metric3D v2 reaches 0.044 AbsRel on KITTI, but it requires known focal length and corrects size globally. It therefore cannot use anchors and does not address spatially varying warp. We include it as an alternative backbone depth, both with and without our layer. If our layer adds nothing on top of it, that would indicate modern metric priors have removed the heteroscedastic error.

Decisive Experiments

The first experiment is a contamination sweep on Dynamic Replica and PointOdyssey. We vary the fraction of anchors placed on moving objects from 0% to 40%, using static-world triangulation for those anchors, and plot background and moving-object AbsRel for every baseline. Fixed-kernel and global methods should degrade with contamination; our method should stay flat. The oracle-masked baseline is the line to beat on moving objects.

The second experiment measures phantom motion. On ground-truth static points, we report the mean magnitude of predicted scene flow before and after alignment, using Any4D and 4RC as backbones. If correcting the warp does not reduce phantom motion, then the claim that warp leaks into motion is false for these models.

The third experiment changes the anchor source on Waymo, from SLAM points to sparse LiDAR beams. We check whether the learned reliability stays high on movers when their anchors are metrically correct, and we report APD to confirm that tracking is not harmed.

Finally, as a sanity check, we reproduce the static ScanNet and KITTI improvement of local over global alignment. This confirms that our implementation of the regression baseline is faithful before any dynamic claim is made. Runtime is reported per frame alongside Any4D's published 0.5 s for 50 frames, so the method stays feed-forward in practice.