When a single moving camera films a street or a room full of people, most 3D reconstruction systems treat the cars and pedestrians as a problem. These objects break the assumption that the world holds still, so they get masked out or down-weighted. This proposal argues the opposite: in monocular video, the moving objects are the best source of absolute size. A single camera cannot tell a small nearby scene from a large distant one, so scale is ambiguous, and today's learned depth models guess that scale with confidence scores that are badly miscalibrated. People and cars have tightly known physical sizes. Rigid cars keep a constant size from frame to frame. Both stand on the static ground, which ties their scale to the background's scale. Together these three facts give a geometric, label-free check on the scale and confidence that a pretrained depth model reports for a particular video. We propose to use that check to recalibrate each scene at test time.
The practical stakes come from the gap between simulated and real data. Dynamic 3D models are trained largely on synthetic video, because only simulation supplies perfect metric depth and motion for moving scenes. That training produces confidence scores that fail on real footage. The constraints we use are physical facts about object size, rigidity and ground contact, not facts about how pixels look. We therefore hypothesize that a correction built on them transfers from synthetic to real much better than confidence learned from appearance. The claim is testable: fit everything on synthetic data, deploy on real driving and indoor video, and measure whether the scene-to-scene scale and calibration errors shrink.
Core Claim
Recent work has left a specific hole. A Calibration Audit of Confidence in Feed-Forward 3D Reconstruction audited seven feed-forward reconstruction networks, among them MonST3R, DUSt3R, MASt3R and VGGT, across 168 settings. It found them overconfident by a median factor of 2.4. For MASt3R, a nominal 90% confidence interval contained the true surface only 0.6% of the time. A two-constant power-law recalibration reduced the median overconfidence to 1.11 on held-out scenes. However, it is fitted per dataset with ground truth, and a residual "missing knowledge of scale" remains scene by scene. Our claim is that this per-scene residual can be estimated without ground truth, from three sources of metric evidence:
- the known size of people and vehicles;
- the constant extent of a rigid body over time;
- contact between dynamic objects and the static ground.
The same estimate also resolves the relative-scale ambiguity between each moving body and the background. Multi-body Depth and Camera Pose Estimation from Multiple Views identified that ambiguity and settled it only by voting against an external depth network (AdaBins), whose own errors pass straight through.
What is new here is not any single cue. Anthropometric priors (Müller et al.) and category shape spaces (Chen et al.) exist already. The new parts are three. First, the cues are used jointly as a test-time calibrator for both scale and confidence of a feed-forward 4D model. Second, the dynamic foreground is treated as signal where prior systems treat it as noise. Third, the calibrator is explicitly tested as a synthetic-to-real transfer mechanism.
Relation to Prior Work
Three systems come closest, and each stops short in a different place.
PRISM-SLAM fuses depth priors from a large pretrained vision model into a probabilistic pose graph through a ray-distance factor, which makes metric scale identifiable online. It handles dynamics by down-weighting pixels whose depth is temporally inconsistent. It therefore discards exactly the objects we want to measure, and it trusts the upstream confidence that the audit shows to be miscalibrated.
MonST3R fine-tunes the decoder of a pointmap network on synthetic dynamic data (PointOdyssey, TartanAir, Spring) plus Waymo, producing per-frame geometry under motion. Its scale and confidence are uncalibrated.
4DGT predicts 4D Gaussians with velocities and lifespans in about 25 ms per frame from real egocentric video. It acknowledges occasional order-of-magnitude scale errors when depth cues are ambiguous.
These three works motivate the proposal: they document the gap. Our method is a layer on top of any of them, not a replacement.
Two other works constrain the design.
- UniDepth gives strong zero-shot single-image metric depth by predicting the camera's ray directions alongside depth. It is the natural source of per-frame metric depth. It is static and per-frame, and it degrades under unusual optics, so we need a check that does not rely on its own confidence.
- Back2Color narrows the appearance gap by painting real-looking texture onto synthetic depth. That attacks how images look, not a model's scene-level scale bias. We use it as a competing way to close the synthetic-to-real gap.
Research Program
The pipeline starts from a pretrained dynamic reconstructor. MonST3R is the primary choice, with UniDepth per-frame depth as a second backbone. It produces per-frame metric pointmaps, raw confidences, and a split into static and moving regions. MonST3R's flow-projection masks supply that split; off-the-shelf instance masks are an alternative.
Each moving instance is lifted into a small per-object track with a category label (person, car, cyclist) and its ground-contact pixels. Contact pixels are the lowest mask pixels for people and the wheel-contact band for cars.
We then solve a small robust problem per scene. The unknowns are three:
- a background scale correction $s$;
- a per-object scale correction $s_k$ for each moving body $k$;
- a per-scene confidence recalibration $(a, b)$ in the power-law form of Deng et al., mapping a raw confidence $c$ to a predicted error standard deviation $\sigma = a\,c^{-b}$.
The objective is
$$ \min_{s,\{s_k\},a,b}\; \sum_k \Big[\rho\big(\tfrac{s_k h_k - \mu_{c_k}}{\tau_{c_k}}\big) + \rho\big(\mathrm{Var}_t(s_k h_{k,t})\big) + \rho\big(\tfrac{s_k z^{\text{obj}}_{k} - s\, z^{\text{bg}}_{k}}{\sigma_k(a,b)}\big)\Big] + \sum_{\text{static } p} \rho\big(\tfrac{r_p(s)}{\sigma_p(a,b)}\big) + \log \sigma_p(a,b), $$
where the symbols are as follows:
- $\rho$ is a robust loss.
- $h_{k,t}$ is the raw metric extent of object $k$ at frame $t$ (height for people, length for cars), and $h_k$ is its time average.
- $\mu_{c}$ and $\tau_{c}$ are the mean and spread of physical size for category $c$.
- $z^{\text{obj}}_k$ and $z^{\text{bg}}_k$ are the object's and the background's depths at the contact pixels.
- $r_p(s)$ is the multi-frame reprojection residual of static point $p$ under scale $s$.
The first term is the size prior. The second is rigidity: it is applied to cars, and to people as a loose bound. The third is ground contact, which couples each body's scale to the background's scale. The last group is a Gaussian likelihood over static pixels that fits the calibration constants from temporal disagreement among repeated observations.
One point matters for honesty. Temporal disagreement alone can only reveal variance, not a bias shared by all frames. The shared scene-scale bias is visible only through the object-size terms, and that is why the moving objects carry the method.
The synthetic-to-real component enters in two places. First, any learned piece is fitted on synthetic data only: Virtual KITTI 2, Dynamic Replica and PointOdyssey, where object sizes, contacts and metric depth are exact. These learned pieces are the per-category size spread $\tau_c$, the contact-band detector, and a small regressor that sets robust-loss thresholds from the observed constraint violations. Second, synthetic data lets us check the key premise directly: that violations of the constraints track true scale and calibration error with the same slope in synthetic and real data. If they do not, the transfer argument fails regardless of benchmark gains.
Baselines
The naive reference point is the backbone left alone. Raw MonST3R or UniDepth metric depth is used at face value, and raw confidence serves as inverse-variance weight in the downstream bundle adjustment. This reference measures how much any correction buys. Deng et al. already report its calibration failure (2.4× median overconfidence) across benchmarks overlapping ours.
The strongest calibration competitor is the power-law recalibration of Deng et al. We run it two ways on the same backbone outputs:
- fitted on the same synthetic sets we use, then applied to real data, which is the fair synthetic-to-real comparison;
- fitted on held-out real scenes with ground truth, which is an oracle-flavored upper reference.
The paper reports median overconfidence of 1.11× on held-out scenes. Beating the synthetic-fitted version on real data counts as a win. Matching the real-fitted version without real labels also counts as a win. We specifically report the per-scene spread of overconfidence, which a dataset-level fit cannot reduce.
The strongest multi-body scale competitor is the KDE-voting alignment of Dal Cin et al. It reports an absolute relative depth error of 0.057 on the KITTI Eigen moving-bodies split, versus 0.089 for DeepV2D. We re-run its scale-alignment step with the same depth prior our method receives (UniDepth rather than AdaBins), so the comparison isolates the alignment rule. Beating it means lower error on moving-body pixels at equal depth prior.
For trajectories we compare against PRISM-SLAM's online metric factor with dynamic gating. It reports 3.04 cm SE(3) absolute trajectory error on TUM fr1/xyz, against 2.86 cm with oracle Sim(3) alignment. That sequence is largely static, and no reported result was found on Lacuna for PRISM-SLAM on Bonn. We therefore run it ourselves on Bonn and TUM dynamic sequences, with our recalibrated confidences replacing its raw ones. This isolates whether the correction helps a metric pose graph.
Ablations remove, one at a time, the size prior, rigidity, ground contact, and the static-variance term. An oracle that gets the true per-scene scale bounds what scale correction alone can reach. The critical ablation removes all object-based terms. If performance stays close to the full method, then moving objects are not doing the work and the core claim is false.
All methods share the same backbone outputs, masks and frames. Every method that has tunable robust-loss thresholds gets an equal tuning budget on synthetic validation data only.
Why not just…?
Why not just take the temporal median of a metric depth model's scale over the video? This removes per-frame scale jitter, and for background depth it is a real threat. It cannot remove a bias that every frame shares, and the scene-level offset Deng et al. describe is exactly that kind of error. It also says nothing about calibrated confidence or about each moving body's scale relative to the background. We include it as a baseline. If it matches our method on scene-scale error, we will report that the object terms help only with confidence and per-body scale, and narrow the claim accordingly.
Why not just recalibrate confidence per dataset, as Deng et al. do? Their fit needs ground-truth geometry from the target domain, and it leaves scene-to-scene residuals. On a new real deployment neither the labels nor a matching dataset exist. The synthetic-fitted variant of their method is the honest competitor. Our experiments measure whether it degrades on real data more than our geometric correction does.
Why not just close the appearance gap, for instance with Back2Color-style translated synthetic data or fine-tuning? Better-looking training data can improve depth, as Back2Color's 0.087 absolute relative error on KITTI (versus 0.093 for MonoViT) shows. It does not by itself give calibrated confidence or a per-scene scale check, and it still leans on stereo during training. We will test it as an alternative backbone under our correction. If translated-data training alone removes the scale bias on real scenes, our test-time step becomes unnecessary for that backbone, and we will say so.
Why not just use an inertial sensor? When accelerometer data exist, visual-inertial fusion (Tanskanen et al.) fixes metric scale with physical units and should win. Our target is the large body of monocular video without such data. Where sequences provide inertial readings we will use them as an upper reference, not a competitor.
Proposed Experiments
Transfer protocol. All learned components are fitted on Virtual KITTI 2, Dynamic Replica and PointOdyssey. Evaluation uses real KITTI, including the Eigen moving-bodies split, plus Bonn RGB-D Dynamic and TUM dynamic sequences. We report five metrics:
- per-scene scale error, measured as the absolute log ratio of estimated to true scale;
- absolute relative depth error on static and on moving pixels separately;
- per-scene overconfidence ratio and 90%-interval coverage, following Deng et al.'s protocol;
- metric object-velocity error on the synthetic sets, where true velocities exist;
- absolute trajectory error on Bonn and TUM when recalibrated confidences feed the pose graph.
First decisive check. On real data, does per-scene overconfidence spread shrink relative to the synthetic-fitted power law? Does median coverage move toward nominal without real labels?
Second decisive check. This one concerns the transfer premise. We regress true scale error on constraint violations separately in synthetic and real data, using KITTI LiDAR as real ground truth. We ask whether the slopes agree. If they do not, the geometric-invariance argument is wrong even if benchmark numbers improve.
Third decisive check. We stratify results by how many people and cars each scene contains. Gains should grow with object count and vanish on object-free scenes, where the method must fall back to the backbone. A flat curve would mean the benefit comes from something other than the moving objects, which would falsify the proposal's central claim.
We will also report failure cases where size priors mislead: children, trucks and unusual vehicles. We will measure how often the robust loss rejects them.
References
- Multi-body Depth and Camera Pose Estimation from Multiple Views
- Reconstructing People, Places, and Cameras
- Learning Canonical Shape Space for Category-Level 6D Object Pose and Size Estimation
- PRISM-SLAM: Probabilistic Ray-Grounded Inference for Scale-aware Metric SLAM
- MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion
- 4DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videos
- UniDepth: Universal Monocular Metric Depth Estimation
- Back2Color: Domain-Adaptive Synthetic-to-Real Monocular Depth Estimation for Dynamic Traffic Scenes
- Live Metric 3D Reconstruction on Mobile Phones