When a single moving camera films a scene and we try to build a 3D map that also records what each thing is, the same point in space often looks different from one frame to the next. There are three different reasons this can happen. The 2D recognition model may have flickered: it called the object "chair" in one frame and "table" in the next. The object may actually have moved. Or the depth estimate from a single image may have put the point in the wrong place, which is a real risk because one image cannot fix distance or absolute size. Today each research community handles only one of these causes. Semantic lifting methods treat all disagreement as label noise and average it away. Dynamic-scene methods treat it as motion or as outliers to delete. Monocular metric SLAM (simultaneous localization and mapping, which tracks the camera while building the map) treats it as depth uncertainty. Each method is therefore wrong whenever the true cause is one of the other two.
We propose to estimate, for every map element, which of the three causes explains its disagreement, and to route each cause to the right fix. Two signals make this separable. Label disagreement has a temporal shape. Recognition flicker alternates back and forth with no order, while real motion produces a single ordered switch: "person" then "floor" at a fixed location, which then follows the person's trajectory. Metric depth from a single-image foundation model, once tied to the static background's scale, gives a direct measurement of motion toward or away from the camera, which optical flow cannot see. The result would be a monocular system that produces a metrically scaled, open-vocabulary 4D map. The more specific scientific claim is that attributing disagreement jointly beats running the three fixes in a cascade.
Core Claim
Multi-view disagreement is evidence to be explained, not a single noise term to be down-weighted. Consider a map primitive $i$, a small 3D Gaussian blob as in Gaussian splatting (a scene representation built from many such blobs, rendered by projecting them to the image). We score three hypotheses $h \in \{F, M, D\}$: recognition flicker, true motion, and depth or scale error. The score comes from the sequence of its observations $o_{i,1:T}$, where each $o_{i,t}$ is the pair (2D label, geometric residual against the map) in frame $t$:
$$ p(h \mid o_{i,1:T}) \propto p(h)\, p(o_{i,1:T} \mid h). $$
Each hypothesis predicts a different kind of observation sequence. Under flicker, geometry stays consistent while labels alternate with no temporal order. Under motion, geometric residuals grow coherently across a group of neighboring primitives, and labels change once, in order. Under depth error, residuals track the depth model's own predicted uncertainty and the viewing angle, while labels stay stable.
The posterior then decides what each primitive is used for. Only flicker-dominated primitives vote in the semantic consensus. Only static, low-depth-error primitives anchor the metric scale fit. Motion-dominated primitives move to a dynamic stream whose depth, including depth along the optical axis, is supervised by metric depth at the shared scale.
Relation to Prior Work
Three static lifting methods motivate the semantic side and supply its machinery. PCF-Lift represents each 3D point's feature as a Gaussian distribution whose spread grows where views disagree. OnlinePG merges segments online by multi-view consensus. UniC-Lift belongs to the same consensus family. All three assume a rigid scene, and PCF-Lift and OnlinePG use posed RGB-D input, so under motion their consensus mechanism averages across physically different objects. We reuse PCF-Lift's probabilistic features inside the flicker branch only.
Two lines of dynamic work motivate the motion side. Split4D uses Gaussian trajectories to define object identity, which is motion informing semantics, and it requires a calibrated multi-camera rig. De4D-SLAM separates static and dynamic Gaussian streams from monocular video using class-agnostic motion probabilities and flow-initialized velocities, but it has no metric grounding and no language labels. We run the arrow in both directions: the ordered structure of label switches becomes evidence of motion.
The closest existing work is Semantic Flow. It learns dynamic semantic fields from monocular video by aggregating optical flow. It works in an arbitrary unscaled frame, treats semantics as an output rather than as evidence, and inherits flow's blindness to motion along the optical axis. We differ on all three points.
Three works supply the metric side. Metric3D v2 gives zero-shot metric depth by mapping images into a canonical camera. PRISM-SLAM treats foundation-model depth as a probabilistic factor so that scale becomes identifiable. S3PO-GS anchors tracking to self-consistent Gaussian pointmaps. All three assume or enforce a static world: PRISM-SLAM's dynamic gating discards moving content, and S3PO-GS's correspondences are corrupted by it. PRISM-SLAM also constrains our design. Its result that depth priors must enter as uncertain factors rather than as ground truth is why the depth-error hypothesis exists at all.
Research Program
The system starts from a monocular Gaussian SLAM backbone with De4D-SLAM-style static and dynamic streams. Every Gaussian carries a three-way attribution posterior, updated over a sliding window of frames. The label likelihood is computed from the per-frame open-vocabulary label sequence projected onto the primitive. Its main statistic is the number of label switches relative to the number of observations: one ordered switch that is spatially coherent with neighbors favors motion, while many alternations favor flicker. The geometric likelihood compares the rendered depth, at the current global scale, to Metric3D v2 depth. The residual is normalized by the depth model's confidence and by viewing angle, so that errors which are view-dependent but temporally incoherent favor depth error.
Metric scale is estimated by a PRISM-style probabilistic alignment. It is restricted to primitives whose posterior mass is on static-with-low-depth-error, which removes the known failure where moving content corrupts scale fitting. With scale fixed, the frame-to-frame change in metric depth of a dynamic group becomes a direct supervision signal for its velocity along the optical axis, alongside flow for sideways motion.
We will also study the failure modes deliberately. These include objects of "movable" classes that sit still (parked cars), objects of "static" classes that move (a pushed chair, an opening door), and slow motion toward the camera. Class-based masking and flow-based motion detection each fail on one of these by construction.
Baselines
The naive reference point runs Metric3D v2 on each frame, back-projects the metric point clouds, and paints each point with that frame's open-vocabulary label. There is no fusion and no motion handling. It shows how much any temporal reasoning buys.
The strongest competitor for scale and tracking in dynamic scenes is De4D-SLAM. It reports 1.95 cm average absolute trajectory error (ATE) on Bonn RGB-D Dynamic. To make it a fair monocular metric competitor, we add a per-sequence global scale taken from the median ratio to Metric3D v2 depth. We then lift semantics onto each of its streams using PCF-Lift's probabilistic fusion, with the same 2D segmenter we use. Beating it means lower ATE under SE(3) alignment, meaning no scale correction at evaluation, which is what tests metric scale, and higher semantic mIoU on dynamic objects. De4D-SLAM's own 1.95 cm figure presumably uses scale-corrected alignment, so we rerun it under both protocols rather than compare to the reported number.
The strongest cascade baseline masks all movable categories with the 2D open-vocabulary segmenter. It then runs PRISM-SLAM for metric tracking, which reports 3.04 cm SE(3) ATE on the static TUM fr1/xyz sequence, and PCF-Lift fusion on the unmasked background. We reuse existing Gaussian primitives for the masked regions per frame.
All systems use the same Metric3D v2 checkpoint, the same 2D segmenter, the same GPU, and matched optimization iterations. Hyperparameters are tuned on one held-out sequence per dataset with an equal number of trials.
Three ablations isolate the key ingredient. The first replaces the three-way posterior with a single learned uncertainty in the style of NeRF On-the-go. The second removes the label-order statistic from the motion hypothesis. The third removes metric depth supervision from the dynamic stream, leaving flow only.
Why not just…?
Why not just mask movable classes with a 2D segmenter and reconstruct the static rest? This is the real threat, and on scenes where only people and cars move it may match us on trajectory error. It fails by construction on the two confounded cases: parked movable-class objects are wrongly deleted, and moved static-class objects are wrongly averaged into the background. Split4D also reports that tracker-dependent decomposition reaches 0.893 vs 0.592 mIoU against a tracking-based method on Multi-Human scenes, which suggests mask errors propagate. The cascade baseline, evaluated separately on those subsets, settles this. If it matches us there, the idea fails.
Why not just use one per-pixel uncertainty to down-weight all disagreement? A single uncertainty variable, as in NeRF On-the-go or RobustSplat, can suppress disagreement but cannot say what to do with it. A flickering chair and a moving person get the same low weight, so the person is lost and the chair's semantics are underfit. The single-uncertainty ablation tests exactly this.
Why not run a dynamic SLAM system, then rescale and label afterwards? Post-hoc rescaling by median depth ratio inherits whatever moving content contaminated the scale fit. It also cannot recover optical-axis motion that flow never registered during optimization. The rescaled De4D-SLAM baseline, plus our per-subset breakdown on motion toward the camera, measures whether joint attribution adds anything over this pipeline.
Proposed Experiments
We need a benchmark with full ground truth for depth, semantics, and motion, so the controlled benchmark is Virtual KITTI 2 (Virtual KITTI 2). It gives ground-truth depth, semantics, and object motion, including cars driving along the optical axis. We run on the monocular front camera. Metrics are attribution accuracy against ground-truth dynamic masks, metric depth AbsRel on static and dynamic pixels separately, velocity error for dynamic objects split into sideways and along-axis components, and rendered semantic mIoU. No reported result was found on Lacuna for any of these tasks on this benchmark.
For real data, we use KITTI-360 (KITTI-360) with the monocular perspective camera. We evaluate depth against LiDAR and 2D semantics on held-out frames for static and dynamic classes. The dataset's own baseline reports 63.5% mIoU for dynamic-object instance segmentation, a different task that we cite only as context. As a single-image reference, Metric3D v2 reports AbsRel 0.044 on KITTI. For tracking we use Bonn RGB-D Dynamic and the dynamic TUM sequences, with RGB input only and depth used solely for evaluation, reporting SE(3) and Sim(3) ATE. For monocular dynamic semantics we use the Semantic Dynamic Scene Dataset, taking one camera as input. There Semantic Flow reports, for example, 0.938/0.703 mIoU on Jumping for multi-scene transfer.
Three checks decide the question. First, on the confounded subsets (parked movable objects, moved static-class objects, motion along the axis), the method should beat the cascade and the single-uncertainty ablation on semantic mIoU and dynamic depth error. Second, the gap between SE(3) and Sim(3) ATE should be smaller than the rescaled De4D-SLAM's on Bonn, showing that scale fitted on attributed-static primitives resists moving content. Third, removing the label-order statistic should measurably hurt attribution accuracy on Virtual KITTI 2. If it does not, semantics adds nothing as motion evidence and the core claim is false.
References
- PCF-Lift: Panoptic Lifting by Probabilistic Contrastive Fusion
- OnlinePG: Online Open-Vocabulary Panoptic Mapping with 3D Gaussian Splatting
- UniC-Lift: Unified 3D Instance Segmentation via Contrastive Learning
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- De4D-SLAM: Gradient-Isolated Static-Dynamic Decoupling for Monocular SLAM in Dynamic Environments
- Semantic Flow: Learning Semantic Fields of Dynamic Scenes from Monocular Videos
- Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation
- PRISM-SLAM: Probabilistic Ray-Grounded Inference for Scale-aware Metric SLAM
- Outdoor Monocular SLAM with Global Scale-Consistent 3D Gaussian Pointmaps
- NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- Virtual KITTI 2
- KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D