Large vision models can already tell you what is in each frame of a video and follow it from frame to frame. A single camera, though, cannot tell you how far away anything is: an object twice as large and twice as far away projects to exactly the same pixels. For still parts of a scene this is a solved-enough problem, because camera motion and modern depth networks pin down the background. For things that move, it is not. A car driving ahead of a moving camera has its own unknown depth in every frame, and the parallax that fixes the background's scale tells you nothing about it. Today's ways of producing 3D or 4D labels without human annotation either use a LiDAR sensor to get depth, or work one frame at a time and inherit the depth network's error. That error grows with the square of distance, which is worst exactly where moving road users matter most.
This note proposes treating each tracked moving object as having a single unknown metric scale that stays fixed for the whole time it is visible. That one number is estimated by pooling three independent kinds of evidence across the track. The first is per-frame network depth, weighted by how uncertain it is along the line of sight. The second is the object's contact with a separately reconstructed, metrically consistent static scene. The third is a weak prior on the size of the object's open-vocabulary class. The result is a camera-only pipeline that turns ordinary video into metric 3D boxes, tracks and language labels. The bet is that most of the remaining error in monocular label transfer for moving objects is a per-object depth bias shared across frames, not per-frame noise and not mask quality. A time-shared scale removes that bias, and independent cues are what make the bias visible at all.
Core Claim
Per-frame metric depth errors on a given moving object are strongly correlated over time: the same object, with the same appearance, at slowly changing range. Averaging them along the track, the obvious fix, therefore reduces variance but leaves a systematic offset. Removing the offset requires evidence whose error is independent of the depth network's read on that object. The most reliable such evidence is where the object touches the static world, because the static world's scale is fixed by camera motion rather than by the object's appearance. The class-size prior serves as a fallback when contact is not visible. The claim is testable in a way that could fail. If a simple temporal median of per-frame depth matches the full method on depth bias at long range, the idea is wrong and the proposal reduces to engineering.
Concretely, let $k$ index a tracked object, $t$ the frames where it is visible, $\hat d_{k,t}$ the depth network's estimate at the object's mask, and $s_k$ a per-object scale correction applied along camera rays. We solve
$$ \min_{s_k,\,\delta_{k,t}} \sum_t \frac{(\log s_k + \delta_{k,t})^2}{\sigma_{k,t}^2} + \sum_{t \in C_k} \frac{\big(s_k e^{\delta_{k,t}} \hat d_{k,t} - d^{c}_{k,t}\big)^2}{\tau_{k,t}^2} + \frac{(\log \ell_k(s_k) - \mu_c)^2}{\rho_c^2} + \lambda \sum_t (\delta_{k,t} - \delta_{k,t-1})^2 . $$
The terms are defined as follows. $\delta_{k,t}$ is a small per-frame residual correction, kept smooth by $\lambda$. $\sigma_{k,t}$ is the line-of-sight depth uncertainty, which grows with range. $C_k$ is the set of frames with visible contact, and $d^{c}_{k,t}$ is the depth at which the object's lowest boundary ray meets the static surface, with uncertainty $\tau_{k,t}$. $\ell_k(s_k)$ is the object's rigid 3D extent implied by the scale. $\mu_c$ and $\rho_c$ are the log-mean and spread of the class-size prior for class $c$. Rigidity enters because one $s_k$ must explain a constant extent in every frame. The form keeps everything in log-depth so that the quadratic growth of error with range becomes a range-dependent $\sigma_{k,t}$ rather than a heuristic cutoff.
Research Program
The pipeline has three stages. Each one reuses an existing component where possible, so that the contribution is isolated in the per-object scale estimation.
The first stage builds the static scene and enforces the static assumption explicitly. It runs a promptable 2D segmenter and video tracker over the clip, removes every mask the tracker marks as possibly moving, and runs monocular dense SLAM with a zero-shot metric depth network on the remainder. The method source is Metric3D v2, which normalizes images to a canonical virtual camera so one network predicts metric depth across cameras. That paper reports that plugging its depth into Droid-SLAM cuts KITTI translation drift from 33.9% to 1.44%. It is the reason a camera-only static map is credible here. The static map is converted to a support surface, and stairs or slopes are kept rather than flattened.
The second stage is the per-object scale solve above, run once per track. This generalizes the terrain-anchoring idea of Crowd4D, which fixes monocular depth for pedestrians by binding their feet to a reconstructed scene surface. Crowd4D is limited to SMPL human bodies and takes roughly four hours per 200 frames. Our version needs only a mask's lowest boundary, applies to any grounded object class, and has one scalar per track to optimize, so it runs in seconds. The line-of-sight uncertainty $\sigma_{k,t}$ follows the observation in USplat4D that monocular dynamic depth uncertainty is strongly anisotropic along the viewing ray. USplat4D uses that uncertainty only to regularize photometric reconstruction. We use it to weight evidence for label transfer. The class-size prior comes from querying a language model for typical dimensions, as in OVM3D-Det. OVM3D-Det trains open-vocabulary monocular 3D detectors from web images by fitting boxes to per-frame pseudo-LiDAR. It treats the prior as near-hard and per-frame, while we treat it as one weak term pooled over a whole track. That difference matters for unusually sized instances.
The third stage emits labels. Once $s_k$ and $\delta_{k,t}$ are fixed, mask pixels are back-projected along rays at corrected depth. A box is fit per frame with OVM3D-Det's surface and containment losses, under the constraint of a shared extent per track. Velocities come from the corrected trajectory. Each track receives one language embedding, averaged over its frames, in the spirit of SAL-4D's temporal CLIP pooling and MonoVoc's one-embedding-per-object storage. Boundary bleeding is handled by rejecting back-projected points whose corrected depth falls outside the object's extent band. This is a geometric test that is possible only once the extent is known, and it replaces OVM3D-Det's fixed mask erosion. The pseudo-labels then train a camera-only monocular 3D detector (Cube R-CNN, as in OVM3D-Det) and a simple 3D tracker.
The closest existing work frames the novelty. Zero-Shot 4D Lidar Panoptic Segmentation (SAL-4D) tracks 2D masklets, projects them onto LiDAR, and distills them into a 4D LiDAR network. UniLiPs fuses foundation-model masks with a LiDAR- and IMU-built static map and separates the points that violate it. Both get every object's depth from the sensor, so per-object scale never arises for them. MonoVoc builds open-vocabulary Gaussian maps from a single camera with one embedding per object, but its SLAM and inversion assume nothing moves. Split4D and WorldTree reconstruct dynamic scenes, but they require multi-view input or carry no semantics. These papers motivate the gap. OVM3D-Det, Crowd4D, USplat4D and Metric3D v2 supply the components. What is new is the estimation target: one metric scale per moving object, recovered from cues with independent errors.
One cited result appears to cut against the proposal. OV3D-Bench concludes that geometric localization in open-vocabulary monocular 3D detection is relatively mature and that semantics is the bottleneck. Its geometric measure is per-frame class-agnostic recall, which tolerates sizable depth offsets and says nothing about trajectory or velocity consistency. We test the disagreement directly by stratifying depth error by range and by whether objects are moving. If moving objects beyond 40 m show no larger bias than static ones, OV3D-Bench is right for our setting too, and the scale solve is unnecessary.
Baselines
The do-nothing reference is per-frame lift-and-accumulate. It runs the same tracker masks and Metric3D v2 depth, back-projects each frame independently, applies OVM3D-Det-style erosion, and fits boxes per frame. It uses identical masks, depth network and box fitting, so any gap reflects the time-shared scale alone.
The strongest naive competitor is temporal smoothing. It takes the same pipeline and replaces $\hat d_{k,t}$ with a robust temporal median along each track, plus a fixed extent per track. This is the reviewer's first suggestion and a real threat. It gets the same tuning budget as our method, with the window and robust loss chosen on a held-out validation split. Beating it means a lower signed depth bias, not only lower variance, in the 40–80 m bin, and a higher 3D AP for the downstream detector.
The strongest existing camera-only method is OVM3D-Det itself, rerun on the front camera of nuScenes and on KITTI with its released configuration and the same depth network, so the gap is not a depth-model upgrade. Its reported novel-class results are 8.2% AP on KITTI and 11.8% AP on nuScenes, against 1.5% and 2.1% for Cube R-CNN with Grounding DINO. A detector trained on our pseudo-labels should exceed these on the same splits and protocol. We report under both the target-aware and the dataset-level prompting protocols of OV3D-Bench, since that benchmark shows the target-aware protocol inflates scores by up to 1.9x.
As an upper reference we run SAL-4D's recipe with real LiDAR. Its paper reports that it reaches 59% of fully supervised performance on SemanticKITTI and improves on prior zero-shot 3D methods by over 5 PQ on Panoptic nuScenes. The question is what fraction of the LiDAR-to-camera gap the scale solve closes. No reported result was found on Lacuna for camera-only 4D pseudo-label tracking on nuScenes, so that number will be established here.
The ablations remove the contact term, remove the size prior, remove time sharing (equivalently, a per-frame $s_k$), and replace the uncertainty weights with uniform ones. These isolate which independent cue removes the bias.
Why not just…?
Why not just take a temporal median of metric depth along each SAM2 track? If depth errors were independent across frames this would be nearly optimal, and the proposal would be superfluous. Our hypothesis is that they are not independent, because the depth network sees the same object with the same appearance cues in every frame. The decisive measurement splits per-object depth error on nuScenes against LiDAR into a track-mean bias and a within-track residual. If the bias is small relative to the residual, the median wins and we will say so. If it dominates, only cues independent of the network, such as contact and size, can remove it. The contact-only and size-only ablations show which cue does.
Why not just run OVM3D-Det per frame and link boxes with a Kalman filter? Per-frame box search pulls every frame toward the language model's canonical size, which biases depth for atypical instances such as vans, trucks and children. A filter on top smooths the positions but inherits that bias, and it cannot use contact with the static scene. This variant is run explicitly, with the same filter used in our tracker. We stratify results by how far each instance's true size, taken from the nuScenes annotations, deviates from the class prior. Our method should matter most in the tails.
Why not just assume a flat ground plane at known camera height? That is cheap and works on flat highways. Crowd4D reports that terrain-aware surfaces improve position scoring by 6.8 PPDS over planar models on VirtualCrowd, and Sheng et al. show that planar heuristics confuse camera tilt with depth. Our contact term already reduces to the plane when the static map is flat, so the comparison is built in. We run a flat-plane variant and report the gap separately on sloped and flat scenes.
Why not just crop proposals from a supervised closed-vocabulary monocular detector and relabel them with SigLIPv2? OV3D-Bench shows this baseline is fast and competitive, but it relies on 3D-labeled training data and does nothing about depth. It answers the semantics question, not ours. We include it only as a semantic reference, and the track-level embedding averaging is compared against per-crop relabeling on the same boxes.
Proposed Experiments
The primary testbed is nuScenes, using front-camera video only. LiDAR and the 3D annotations are used purely for evaluation. The headline metrics fall into three groups. The first is signed and absolute per-object depth error, binned at 0–20, 20–40, 40–80 and above 80 m and split into moving and static objects. The second is box-level 3D AP for a Cube R-CNN trained on the pseudo-labels, under both prompting protocols. The third is velocity error and tracking quality on moving tracks. KITTI repeats the detection comparison against OVM3D-Det's reported 8.2% AP, and Metric3D v2's static-scene Abs Rel of 0.044 on KITTI serves as context for how much worse dynamic-object depth is. ARKitScenes, where OVM3D-Det reports 19.7% AP on novel classes, tests whether contact with furniture and floors helps indoors. VirtualCrowd checks the human special case against Crowd4D's reported 91.43 PPDS, at a fraction of its compute.
Two checks are decisive. The first is the bias-versus-variance decomposition: the idea stands only if track-mean bias for moving objects beyond 40 m is a large share of their error and our method removes it while the temporal median does not. The second is the cue-independence test. On tracks with no visible contact, the gain should shrink toward what the size prior alone provides, and on atypically sized instances the size prior should hurt unless contact is present. If both patterns hold, per-object scale is the right unit for monocular 4D label transfer. If neither holds, the evidence supports OV3D-Bench's view that geometry is not the bottleneck.
References
- Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation
- Crowd4D: Scene-Aware Monocular 4D Crowd Reconstruction
- Uncertainty Matters in Dynamic Gaussian Splatting for Monocular 4D Reconstruction
- Training an Open-Vocabulary Monocular 3D Detection Model without 3D Data
- Zero-Shot 4D Lidar Panoptic Segmentation
- UniLiPs: Unified LiDAR Pseudo-Labeling with Geometry-Grounded Dynamic Scene Decomposition
- MonoVoc: Decoupling Geometry and Semantics for Lightweight Monocular Open-Vocabulary 3D Gaussians
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- WorldTree: Towards 4D Dynamic Worlds from Monocular Video using Tree-Chains
- OV3D-Bench: A Diagnostic Benchmark for Open-Vocabulary Monocular 3D Detection