Networks that rebuild a moving 3D scene from a short video in one pass, with no per-scene optimization, now exist. They still describe geometry the way stereo matchers did decades ago: each pixel gets one depth value, and 3D points are produced by pushing each pixel out along its camera ray. This is a per-pixel version of the fronto-parallel assumption, the belief that every small image patch faces the camera squarely. It causes known errors. Roads, ramps and car hoods come out as staircases, and foreground objects "fatten" into the background at their edges. In a dynamic scene the damage grows, because a per-pixel motion estimate is then applied to points that were placed wrongly. The mistakes show up as tearing and shearing at the edges of moving objects.
We propose to replace per-pixel depth and per-pixel motion with small image tiles. Each tile carries a tilted plane (a depth plus two slopes) and one rigid 3D motion, and the tiles are refined by warping features across time. A plane moving rigidly maps exactly between frames through a homography, so matching between frames becomes geometrically correct on slanted and rotating surfaces. The same tiles also carry a static-versus-moving estimate. That estimate lets us recover metric scale (real units, not up-to-scale) from sparse depth anchors, such as a few LiDAR beams, while ignoring anchors that land on moving objects. Those anchors are exactly the ones that corrupt the global least-squares fits used today.
Core Claim
In feed-forward 4D reconstruction, a joint slanted-plane-plus-rigid-motion parameterization at tile level lowers geometry and scene-flow error on slanted surfaces and at motion boundaries. The central test is whether it beats a per-pixel-depth model that is simply trained with a surface-normal loss. If it does not, the parameterization is not the reason for any gain. As a secondary claim, solving for scale in closed form using only confidently static tiles is more robust to moving-object contamination of sparse metric anchors than a learned global scale token or a plain least-squares fit.
Prior Work and Where This Sits
Three recent feed-forward dynamic models motivate the gap. Any4D predicts per-pixel depth, camera poses, scene flow relative to the first frame, and a learned global metric scale token. 4RC encodes video with time tokens and queries a displacement field on top of reference-frame depth. BTimer outputs a Gaussian-splat scene frozen at a chosen time. All three lift geometry from per-pixel depth maps and never represent local surface slant. Any4D also ties flow to the first frame, so objects that appear later cannot be tracked. We keep their transformer backbone and change what the heads predict and how the output is refined.
The method comes mostly from stereo. HITNet predicts 4×4 tiles that each hold a disparity, two disparity gradients and a learned descriptor. It refines them by 2D propagation and in-network warping instead of 3D-convolved cost volumes, and reaches SceneFlow EPE 0.36. We carry that tile hypothesis from a calibrated stereo pair to a video with an unknown moving camera, and we add a per-tile rigid motion. Classical work explains why this matters. Kuschk and Psota et al. document staircase artifacts. Jung et al. document boundary fattening. Zhan et al. show that learned monocular depth inherits the same bias.
Two papers already predict slanted geometry feed-forward, but only for static scenes. PLANA3R predicts metric planar primitives from two unposed static views (ScanNetV2 F-score 92.52). RoadBEV reconstructs road elevation in a bird's-eye grid. Both show that slanted primitives help, and both break under motion. Our tiles are denser than PLANA3R's primitives and are defined per frame, which is what lets them move.
One result constrains the design. Revisiting Depth Representations for Feed-Forward 3D Gaussian Splatting reduces depth-discontinuity artifacts with a point-map loss from an external model. It gains at least 2 dB but costs about 65 ms and 6 GB per alignment, and it depends on a static-scene teacher. A loss-only fix is therefore a real competitor, and we test it directly.
For scale, LiDARTouch shows that about four LiDAR beams suffice to fix global scale. Wofk et al. fit scale to sparse visual-inertial points by least squares and then refine the result. Both assume the anchors lie on static structure. Metric3D v2 instead obtains metric depth by normalizing focal length, which requires known intrinsics. We take the sparse-anchor route and make it motion-aware.
Research Program
The backbone is an alternating-attention transformer over $N$ frames in the style of Any4D, trained on the same mixture Any4D reports: Kubric, PointOdyssey, Dynamic Replica, Virtual KITTI 2 and Waymo. At 1/4 resolution, a tile head predicts the following for each tile $k$ in frame $t$:
- a depth plane $z(u,v) = z_k + a_k (u-u_k) + b_k (v-v_k)$, where $(u_k, v_k)$ is the tile centre, $z_k$ the centre depth and $(a_k, b_k)$ the image-space depth slopes;
- a rigid motion $(R_k, \mathbf{t}_k)$ to frame $t+1$, which is the identity for static tiles;
- a static probability $\pi_k$;
- a confidence;
- a descriptor.
Because each tile is a plane that moves rigidly, it induces an exact map from frame $t$ to $t+1$. Refinement follows HITNet. At each of a few iterations, neighbour-frame features are warped through that map, composed with the predicted camera motion. A matching cost is computed, and a small network updates the tile parameters and propagates hypotheses between neighbouring tiles so that edges can snap to the correct side. Dense depth and flow are then read off each tile's plane analytically, so no staircase can form inside a tile.
Motion is predicted between adjacent frames and chained. This avoids Any4D's first-frame dependency. Chaining can accumulate drift, so we report long-horizon tracking honestly.
Metric scale is solved in closed form. Let $\hat z_i$ be the network's up-to-scale depth at sparse anchor $i$, $z_i$ the measured metric depth, and $w_i = \pi_{k(i)} c_{k(i)}$ the static probability times the confidence of the tile containing the anchor. The scale is
$$s^\ast = \arg\min_s \sum_i w_i \, \rho\!\left(s \hat z_i - z_i\right),$$
where $\rho$ is a robust loss. This scale is differentiable and used during training. Anchors are compared against the tile's plane rather than a single pixel, which reduces errors from anchors falling on the wrong side of a fattened edge. Without anchors, the model falls back to a learned scale token, as in Any4D.
Baselines
The naive reference is the same backbone and training data with per-pixel depth and per-pixel flow heads, and scale taken from an unweighted least-squares fit to the anchors (Wofk-style). This isolates everything we add.
The strongest existing method is Any4D. It is the only feed-forward model in the file that combines multi-frame dynamics with metric scale, and it reports dense scene-flow EPE 1.13 on Kubric-4D with a dynamic camera, against 2.44 for St4RTrack. We run it with released weights if they exist. We also retrain it on our data mixture and step budget so that the head is the only difference. Beating it means lower EPE than 1.13 on that split and lower error in the slope-stratified and boundary strata defined below, with no loss on flat, front-facing regions.
4RC is the second dynamic baseline. It reports a 36% dense-tracking APD gain over V-DPM on Waymo, but the file gives no absolute Waymo value. We therefore compare against numbers we measure ourselves under identical settings.
For per-frame metric depth, the reference is Metric3D v2, run with ground-truth focal lengths, which is favourable to it. It reports KITTI AbsRel 0.044.
Two ablations isolate the key ingredient. The first is the naive model plus a depth-normal consistency loss, in the spirit of Metric3D v2's joint depth-normal refinement. The second is the naive model plus the point-map loss of Revisiting Depth Representations. A third ablation keeps slanted tiles but assigns fronto-parallel motion (translation only, no rotation). It separates the contribution of slant from that of rigid tile motion. For the scale study, the baselines are the Any4D scale token, unweighted least squares, and the median of depth ratios after masking movers with a 2D segmenter.
All variants share backbone, data, iteration budget and hyperparameter search budget.
Why not just…?
Why not just add a surface-normal loss to per-pixel depth? This is the most serious threat, and it may be enough. Metric3D v2 already shows that joint depth-normal reasoning helps on static images. What a loss cannot do is change the warp used to match frames. A per-pixel head still matches small patches as if they were front-facing and moved without rotating. The first ablation settles this. If it matches the tile model on sloped surfaces and at motion boundaries, we drop the parameterization claim.
Why not just run Metric3D v2 per frame plus an off-the-shelf tracker? Any4D reports that decoupled pipelines of this kind accumulate errors and are much slower: 11.56 s against 0.50 s for 50 frames with SpatialTrackerV2. Metric3D v2 also needs intrinsics. We still include it as a per-frame depth reference, because on static regions of KITTI it may win.
Why not just take the median depth ratio after masking movers with a segmenter? This may be enough for scale when masks are good. Split4D notes that missed frames or mislabelled boundaries propagate errors, and a class-based mask cannot catch parked-then-moving cars or anchors lying on an object's edge. The anchor-corruption experiment below measures exactly this.
Why not just use dense 3D cost volumes? HITNet shows that tile hypotheses with warping match or beat cost-volume stereo at far lower latency. Over $N$ frames, a volume per frame pair multiplies that cost. We report runtime so readers can judge the trade-off.
Proposed Experiments
The main evaluation covers several benchmarks. Kubric-4D measures scene-flow EPE against the 1.13 reported for Any4D. Virtual KITTI 2 measures depth and scene flow, including its 15° and 30° rotated-camera variants, where road slant relative to the viewing ray grows. No Any4D number on Virtual KITTI 2 is reported in the file, so we measure it. Waymo DriveTrack measures dense 3D tracking APD. KITTI measures metric depth AbsRel, with Metric3D v2's 0.044 as the reference.
Every error is also reported in two strata. The first bins pixels by the ground-truth angle between surface normal and viewing ray. The second covers pixels within a few pixels of motion boundaries. A gain that appears only in aggregate does not count.
As a stretch test of slope fidelity, we rasterize our reconstructions of the RSRD road dataset into elevation maps. We compare them with the 1.83 cm reported for RoadBEV-mono, knowing that RoadBEV is specialized for roads and may remain better.
The scale study simulates four-beam LiDAR on KITTI and Waymo, following LiDARTouch, and injects increasing fractions of anchors onto moving objects. It measures metric depth error as the fraction grows. The decisive result is a flatter degradation curve for the static-weighted solve than for Any4D's scale token, least squares, and masked median.
The central decisive check is the comparison with the normal-loss ablation in the high-slant and motion-boundary strata of Kubric-4D and Virtual KITTI 2. If the slanted-tile model does not beat it there, the claim is wrong.
References
- Any4D: Unified Feed-Forward Metric 4D Reconstruction
- 4RC: 4D Reconstruction via Conditional Querying Anytime and Anywhere
- Feed-Forward Bullet-Time Reconstruction of Dynamic Scenes from Monocular Videos
- HITNet: Hierarchical Iterative Tile Refinement Network for Real-time Stereo Matching
- Fast and Accurate Large-Scale Stereo Reconstruction Using Variational Methods
- MAP Disparity Estimation Using Hidden Markov Trees
- Stereo reconstruction using high order likelihood
- Self-supervised Learning for Single View Depth and Surface Normal Estimation
- PLANA3R: Zero-shot Metric Planar 3D Reconstruction via Feed-forward Planar Splatting
- RoadBEV: Road Surface Reconstruction in Bird's Eye View
- Revisiting Depth Representations for Feed-Forward 3D Gaussian Splatting
- LiDARTouch: Monocular metric depth estimation with a few-beam LiDAR
- Monocular Visual-Inertial Depth Estimation
- Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation