Neural networks can now look at two photos of a scene and directly predict a 3D position for every pixel. Such a prediction is called a pointmap. This has made camera tracking and 3D reconstruction from casual video far easier. When things in the video move, however, the methods that stitch many such predictions into one consistent reconstruction still have to decide which pixels belong to the static world, since only static pixels can anchor the camera. Today that decision is usually made once, by a separate 2D motion estimator, and then treated as fixed. Every mistake in that label is passed straight into the camera trajectory. We propose to make the static-or-moving label a variable that the pose-and-geometry optimizer estimates jointly with the cameras. It would start from motion that a dynamic pointmap network predicts directly in 3D, and it would be revised as the reconstruction improves.
The core claim is that under this design, final camera error becomes largely insensitive to errors in the initial motion estimate, while pipelines with fixed labels degrade steadily as those errors grow. We also claim that the camera stays constrained even when most of the image is moving, because moving pixels still contribute motion-compensated constraints instead of being discarded. Both claims can be measured directly. We propose a "cascade sensitivity" protocol that deliberately corrupts the upstream motion estimate and plots the resulting pose error. We then test on the standard dynamic-scene tracking benchmarks.
Where the Gap Is
Four lines of work meet here. Direct 3D regression came from DUSt3R, which predicts dense pointmaps and confidences from unposed image pairs. Differentiable pose refinement treats camera poses as parameters optimized by gradient descent against image evidence. BARF showed that a coarse-to-fine schedule widens the region from which optimization converges. Look Gauss, No Pose made pose refinement fast inside Gaussian splatting, a representation of the scene as many small 3D blobs. Both assume a static scene: moving content produces wrong pose gradients. Work on cascaded errors shows that a downstream 3D stage cannot recover what an upstream 2D stage got wrong. View Invariant 3D Human Pose Estimation shows this for discarded visual cues, and Accurate Localization of 3D Objects from RGB-D Data Using Segmentation Hypotheses shows that 3D recall is capped by 2D detector recall. Finally, dynamic reconstruction must cope with the static-scene assumption breaking.
The closest systems each cover part of this ground, and each motivates the proposal. MonST3R fine-tunes DUSt3R on dynamic data and runs a global optimization over cameras and pointmaps. Its static mask comes from comparing the flow that camera motion alone would produce against flow from an off-the-shelf 2D optical flow network. This is exactly the fixed upstream dependency that the cascade literature warns about, and it is our main target.
Dynamic Point Maps (DPM) goes further and supplies the method we reuse. For a pair of frames it predicts where each pixel is at both timestamps, in one shared camera frame, so 3D scene flow falls out by subtraction with no 2D flow. It is purely feed-forward, however. Its authors report that rotation errors remain high under large object motion and that accuracy drops under motion blur.
POMATO and Easi3R also obtain dynamic masks without 2D flow. POMATO uses a 3D matching head. Easi3R reads masks from the decoder's attention statistics without any training. Neither jointly optimizes poses and geometry at test time, and both treat the mask as a fixed product of a forward pass. So "motion masking without 2D flow" exists. What does not exist is an optimizer that treats the mask as something to be estimated.
Back on Track (BA-Track) supplies a key idea: it keeps moving points in bundle adjustment by subtracting their object motion, making them "pseudo-static." It does this with a sparse point tracker and monocular depth, and if the tracker loses a point, the error cascades into the optimization. Joint Optimization for 4D Human-Scene Reconstruction in the Wild (JOSH) shows that joint optimization beats separately run stages on human-scene capture, but it depends on its initializers and runs slowly. Split4D and RobustSplat remove 2D trackers from dynamic Gaussian reconstruction, but both assume the cameras are already posed.
The contrast is therefore specific. Prior work either optimizes cameras with a fixed mask (MonST3R), predicts masks with no optimization (DPM, POMATO, Easi3R), or keeps moving points through a sparse tracker (BA-Track). We optimize cameras, dense geometry and per-pixel motion labels together. The dense motion compensation comes from DPM's two-timestamp pointmaps rather than from a tracker.
Core Mechanism
For each frame pair $(i,j)$, a DPM-style network predicts two pointmaps, both expressed in camera $i$'s coordinates. The first, $A_{ij}(u)$, gives the 3D position of pixel $u$ of frame $j$ at frame $j$'s own time. The second, $B_{ij}(u)$, gives the same surface point at frame $i$'s time. For a static pixel the two coincide. For a moving pixel their difference is its predicted scene flow.
Unknowns are optimized on the SE(3) tangent space, following the Lie-group parameterizations used across the pose-refinement literature. They are world camera poses $T_i$, per-pair scales $s_{ij}$, small per-pixel geometry corrections, and a soft label $w_u \in [0,1]$ saying how static pixel $u$ is. The energy is
$$E=\sum_{(i,j)}\sum_{u}\Big[w_u\,\rho\big(\|T_i s_{ij}A_{ij}(u)-\bar X_u\|\big)+\lambda\,\rho\big(I_i(\Pi_i(s_{ij}B_{ij}(u)))-I_j(u)\big)\Big]+\mathcal{R}(w).$$
Here $\bar X_u$ is a single world point shared by all pairs in which that surface is seen. $\rho$ is a robust penalty, $\Pi_i$ projects a point into image $i$, $I_i$ is image $i$'s intensity, $\lambda$ weights the photometric term, and $\mathcal{R}$ is a prior that pulls $w$ toward its initial value, which is set from the magnitude of the predicted scene flow $\|A_{ij}-B_{ij}\|$.
The first term is the strong constraint that only static pixels deserve: one world point across the whole sequence. This is what anchors long trajectories against the scale drift that feed-forward stitching suffers. The second term applies to every pixel, moving or not. The motion-compensated point, projected into the other image, must land on matching colour.
The two terms together are the anti-collapse argument. Declaring everything "moving" does not free the cameras, because the photometric term still ties them to the images through $B_{ij}$. Declaring a truly moving pixel "static" is penalized by the multi-frame term. The labels are updated by comparing the two per-pixel explanations, in the style of expectation-maximization. The photometric term is scheduled coarse-to-fine, as in BARF, to keep the region of convergence wide. Everything runs over a sliding window, then a sparse global pass over keyframes.
The honest risk is that $B_{ij}$ is exactly where DPM is weakest: large motion and blur. The photometric term therefore carries its own robust weight, so unreliable scene flow is downweighted rather than trusted.
Research Program
The work has three parts. First, we build the optimizer on top of a released DPM-style backbone, without retraining, so that any gain is attributable to the refinement stage. Second, we define and run the cascade sensitivity protocol. On synthetic sequences with ground-truth motion, namely PointOdyssey and Kubric MOVi-G and MOVi-F, we corrupt the initial motion estimate in controlled ways. We flip a fraction $p$ of labels, dilate or erode object boundaries, and drop whole objects. We then plot absolute trajectory error against $p$. We expect fixed-label pipelines to degrade roughly in proportion to $p$ near small $p$, since each mislabeled pixel contributes a biased residual. We expect our curve to stay flat until errors are large enough that the prior $\mathcal{R}$ dominates. The protocol turns the cascade literature's qualitative warning into a measured quantity. Third, we evaluate on the real benchmarks.
Baselines
The naive reference point is DPM run feed-forward, with pairwise predictions chained into a trajectory and no optimization. This tells us what refinement adds at all. The strongest feed-forward competitor is POMATO. It reports a 55.4% reduction in relative rotation error on TUM-dynamics over prior methods, but the file gives no absolute value, so no reported result is found on Lacuna for an absolute POMATO TUM number. We will rerun it on our splits.
The strongest optimization-based system in our direct regime is MonST3R's global optimization, with its 2D-flow-derived mask. The file gives no numeric Sintel ATE for it; the paper states only that it achieved the best camera translation and rotation accuracy there. So no reported result is found on Lacuna for MonST3R ATE, and we will rerun it with the released configuration.
The strongest reported pose number in the file comes from a different family. BA-Track reports ATE 0.034 on Sintel, against 0.089 for LEAP-VO and 0.175 for DROID-SLAM (Back on Track). On DyCheck, Easi3R on DUSt3R reports ATE 0.021 against 0.035 for plain DUSt3R (Easi3R). Beating the proposal's bar means matching or beating 0.034 ATE on Sintel and 0.021 on DyCheck under the same evaluation protocol. On TUM-dynamics, Bonn and AirDOS Shibuya, where the file reports no numbers, it means beating our reruns of all baselines.
Two ablations isolate the key ingredient. The first, "fixed native mask," runs our exact optimizer but freezes $w$ at its initial DPM-derived value. The second, "flow mask," freezes $w$ at MonST3R's flow-derived mask. A third ablation drops the photometric term, so that moving pixels are simply filtered out, which tests the claim that retaining them matters.
All methods get the same frames, intrinsics policy, window length and GPU. Every optimization-based method gets an equal wall-clock budget and an equal hyperparameter search of the same size on a held-out Sintel split.
Why not just…?
Why not just swap MonST3R's optical-flow mask for a POMATO or Easi3R mask? This is the most serious threat, and the "fixed native mask" ablation is designed to settle it. If a better fixed mask closes most of the gap, then the contribution reduces to "use 3D-native masks," and we would report that. Our prediction is narrower. Native masks still fail on specific regimes, such as blur and objects that start or stop moving. On the cascade curves a fixed mask of any origin should degrade with $p$, while jointly estimated labels should not. If the curves overlap, the idea fails.
Why not just use a robust loss and no mask at all? A robust penalty in MonST3R-style alignment silently discards whatever disagrees. That is the filtering stance of RobustSplat, and the file's discussion of Back on Track notes that it breaks when moving content dominates the frame. The no-photometric ablation, evaluated separately on AirDOS Shibuya crowd sequences and dynamic-heavy TUM sequences, measures this directly. On mostly static sequences a robust loss may well tie us, and we will say so.
Why not just use BA-Track, which already keeps moving points? BA-Track is strong, with the best ATE reported in the file. Its failure mode is losing tracks under occlusion, and it optimizes only sparse tracks plus a coarse depth-scale grid. Our dense two-timestamp pointmaps need no tracks to survive across frames. We compare on Sintel and on the long-occlusion subsets of PointOdyssey, where the tracker survival rates reported by PointOdyssey (PIPs++ at 50.47%) show that tracks do get lost. If BA-Track still wins on Sintel, our contribution shrinks to robustness under occlusion, and we would claim only that.
Why not just stay feed-forward, since POMATO is fast? Feed-forward stitching drifts in scale over long sequences, and DPM's authors report high rotation error under large motion. We report the accuracy–runtime tradeoff honestly: seconds per sequence against POMATO's reported 0.7 seconds for six frames. We also test whether a short refinement budget already recovers most of the gain.
Decisive Checks
The idea stands or falls on three results. The first is flat cascade-sensitivity curves on PointOdyssey and Kubric, where the fixed-mask ablations are not flat. The second is ATE on Sintel at or below 0.034 and on DyCheck at or below 0.021 under the shared protocol. The third is a clear advantage over the no-photometric ablation on the sequences where moving content dominates the frame. Secondary checks are depth absolute relative error on Sintel and Bonn, relative pose error on TUM-dynamics, and dynamic-mask quality on DAVIS, since the jointly estimated labels are themselves a segmentation. If the first check fails, the claim that labels should be jointly estimated is wrong, whatever the benchmark numbers show.
References
- DUSt3R: Geometric 3D Vision Made Easy
- BARF: Bundle-Adjusting Neural Radiance Fields
- Look Gauss, No Pose: Novel View Synthesis using Gaussian Splatting without Accurate Pose Initialization
- View Invariant 3D Human Pose Estimation
- Accurate Localization of 3D Objects from RGB-D Data Using Segmentation Hypotheses
- MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion
- Dynamic Point Maps: A Versatile Representation for Dynamic 3D Reconstruction
- POMATO: Marrying Pointmap Matching with Temporal Motion for Dynamic 3D Reconstruction
- Easi3R: Estimating Disentangled Motion from DUSt3R Without Training
- Back on Track: Bundle Adjustment for Dynamic Scene Reconstruction
- Joint Optimization for 4D Human-Scene Reconstruction in the Wild
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking