A single camera cannot tell a small nearby scene from a large distant one. Current systems that reconstruct people moving through the world from ordinary video handle this by asking a pretrained depth network for meters, then smoothing the result over time. That approach discards a strong constraint. The true scale of a video is one number, or one that changes slowly, and the video contains a ruler that never changes size: the person's own skeleton. When that person's foot is planted, it also touches a surface whose depth the scene model can measure. We propose to estimate scale as a single slowly varying quantity tracked causally over the video. Three sources of evidence feed it: the body's bone lengths, the depth network's metric guess, and foot–ground contacts. The probabilities from a semantic segmenter decide how much each piece of evidence counts in each frame.
The segmenter matters because every scale cue here passes through one. Height-based calibration such as Single View Physical Distance Estimation using Human Pose solves focal length and metric distance from standing humans in a single frame. It breaks when a person sits, is cut off by an occluder, or when a mask leaks onto a bag or a second person. Contact-based grounding breaks when a low obstacle is mistaken for walkable floor. These failures are not independent per frame. A person sitting for twenty seconds or a pole occluding the legs for a whole walk-by biases every frame in the same direction. Robust per-frame filtering, which assumes outliers are scattered, therefore misses them. Our claim is that turning segmentation class probabilities into per-measurement noise inside a temporal scale filter gives metric trajectories that are more accurate and degrade more gracefully under segmentation errors than either foundation-model depth with smoothing or robust aggregation of per-frame anatomical scale.
Core Claim
Treat the log metric scale $s_t$ of the video's relative depth and camera trajectory as the state of a one-dimensional causal filter. Each frame supplies measurements $z_{t}^{k}$ of $s_t$ from three sources $k$:
- Anatomical: the ratio between the person's bone lengths in the reconstruction's units and a person-specific metric bone length estimated online.
- Foundation depth: the scale implied by a metric depth model.
- Contact: for a foot predicted to be in stance, the ratio between its body-derived metric depth and the relative scene depth at its pixel.
Each measurement enters with variance
$$\sigma_{t,k}^{2} = \frac{\sigma_{k}^{2}}{\pi_{t,k}},$$
where $\sigma_k^2$ is a base noise level fitted on training data. The reliability $\pi_{t,k}\in(0,1]$ comes from segmentation probabilities. For the anatomical measurement, it is the mean "person-instance" probability over the pixels the relevant limbs project onto, multiplied by an upright-posture score. For the contact measurement, it is the product of contact probability and the segmenter's "walkable surface" probability at the foot pixel. A heavy-tailed mixture term keeps confidently wrong labels from dominating. The update is then a standard precision-weighted fusion with a small process noise on $s_t$, so scale can drift slowly, for example when a zoom changes the focal length, but cannot flicker.
What is new is not any single cue. Prior systems already use each cue separately. The new parts are three. First, contact ties the human's metric scale to the scene's relative depth. Second, scale is treated as a shared per-video state rather than a per-frame output. Third, segmentation errors propagate as noise inflation rather than hard masks.
Relation to Prior Work
Three systems motivate and constrain the design. Depth-Guided Metric-Aware Temporal Consistency for Monocular Video Human Mesh Recovery is the closest. Its D-MAPS module averages bone lengths over time, weighted by depth confidence, to keep the body from shrinking or growing, and it fuses depth-model features with RGB through confidence gating. It works in camera coordinates on a single subject, its weights come from depth features rather than semantic class evidence, and it does not ground the scene or camera trajectory. We reuse its observation that temporal bone-length aggregation stabilizes body scale. We extend it in two ways: the stabilized body becomes a ruler for the world, and the segmenter decides when the ruler is trustworthy.
OnlineHMR supplies the causal setting and our host system. It runs a streaming mesh-recovery branch beside an incremental SLAM branch that hard-masks humans with SAM2. It takes metric scale from MoGe-V2 with exponential moving average smoothing. Its scale is therefore only as good as an ungrounded depth network. We replace that step while keeping its causal structure.
Crowd4D shows that forcing feet onto a reconstructed non-planar interaction surface resolves depth ambiguity in crowds. It does so offline, taking 4 hours for 200 frames with 100 people, and without checking whether the surface is semantically walkable. We borrow the contact-as-scale idea in a causal, single-scalar form.
On the semantic side, VSO is our methodological precedent. It keeps class distributions per landmark and uses them in a soft reprojection term, cutting monocular translational drift by up to 45% on KITTI relative to ORB-SLAM2. That is direct evidence that soft semantic evidence beats hard labels for scale drift. VSO assumes static classes, however, and cannot use people. We do the opposite: the person is the main metric anchor.
Contact and Human Dynamics from Monocular Video supplies the contact predictor design and a constraint. Full centroidal-dynamics optimization takes 30 to 60 minutes per 2 seconds of video and assumes flat ground. We therefore use contact only as a scale measurement, not as a physics solve.
Metric3D v2 is the strongest single-frame metric depth model in the file. Its reliance on known focal length and its lack of temporal coupling are exactly the gaps our filter targets.
Research Program
The system has four parts, all of which run causally. The first is a host pipeline: OnlineHMR's streaming body branch and SLAM branch, with MoGe-V2 or Metric3D v2 producing relative scene depth and a raw metric guess. The second is a panoptic segmenter that produces per-pixel class probabilities for person instances and for surface categories (floor, road, sidewalk, stairs, versus furniture, curb top, vehicle). These probabilities are used in two places. They weight the SLAM features softly, VSO-style, replacing OnlineHMR's binary mask. They also produce the reliability terms $\pi_{t,k}$ above.
The third part is the online person-specific metric skeleton, which bootstraps the ruler. A population prior on bone lengths, as in Fei et al., is gradually replaced by an estimate accumulated from high-reliability frames, where the person is upright, unoccluded and confidently segmented. The fourth part is the scale filter itself, which feeds $s_t$ back to rescale the camera trajectory and human root translation.
The analysis has two goals. The first is to measure how segmentation error turns into scale error. We will inject controlled corruptions into the segmenter outputs, either independent per frame or temporally persistent: limb occlusion masks, mask leakage onto adjacent people or objects, and floor-versus-obstacle label swaps. We will then plot metric scale error against corruption rate for each fusion rule. The second goal is to identify which measurement source carries scale in which regime. We will log each source's posterior weight over time across walking, sitting, and stair sequences.
Baselines
The naive reference point is OnlineHMR as published: MoGe-V2 metric scale with exponential moving average smoothing, run with identical body and SLAM branches so that only the scale step differs. A second reference swaps in Metric3D v2 under the same smoothing, with focal length supplied where the dataset provides intrinsics and estimated otherwise. Metric3D v2's single-image accuracy is high: Abs Rel 0.044 on KITTI and 0.047 on NYUv2, per Metric3D v2. Beating it on video scale would therefore be meaningful. For OnlineHMR on EMDB, the file states that it outperforms prior online methods in world-aligned joint error, but no reported number was found on Lacuna for OnlineHMR's WA-MPJPE. We will rerun it ourselves on identical splits.
The strongest competitor on body-scale stability is the D-MAPS idea from Depth-Guided HMR. That paper reports MPJPE of 69.31 mm on 3DPW. We will reimplement its depth-confidence-weighted bone-length averaging as the anatomical source inside our filter, with no semantic weights and no contact, under the same compute and tuning budget. This isolates whether semantic reliability and the contact link add anything beyond depth-confidence weighting.
For distance estimation, the anatomical-only baseline is Fei et al.'s closed-form method with RANSAC. It reports 78% four-bin distance accuracy on MEVADA against 40% for a ResNet baseline, per Single View Physical Distance Estimation using Human Pose. We will run it per frame and with a temporal median.
Three ablations isolate the key ingredient:
- the full filter with all $\pi_{t,k}=1$, which keeps temporal fusion but removes semantics;
- the filter with hard thresholded masks in place of probabilities;
- the filter without the contact source.
Beating the baselines means two things. On EMDB, metric trajectory scale error and world-aligned joint error must be lower than those of the best smoothed foundation-depth baseline, with non-overlapping bootstrap confidence intervals over sequences. Under persistent segmentation corruption, the soft-weighted filter's scale error must grow significantly more slowly than the temporal-median and hard-mask variants.
Why not just…?
Why not just trust a metric depth foundation model and smooth it? This is the real default, and OnlineHMR shows it already gives competitive online world-grounded results. The weakness is that the per-frame metric guess depends on intrinsics, as Metric3D v2 itself states. Smoothing removes flicker but not bias, and on web or surveillance video without metadata the bias can be large. Crowd4D also reports that unconstrained monocular depth on non-planar scenes leaves people floating or sinking. Our experiments settle the question directly. If the smoothed foundation-model baseline matches us on EMDB and on intrinsics-withheld clips, the anatomical and contact sources add nothing, and we would report that.
Why not just take a robust median of per-frame human-height scale? This is the alternative that could genuinely match us, because medians reject scattered outliers well. It fails exactly when segmentation errors are temporally correlated, as in long sitting stretches or persistent occluders. In those cases more than half the frames can be biased the same way. The corruption study separates the two cases. If the median matches the soft filter under persistent corruption, we will drop semantic weighting from the anatomical source and keep only the contact link. That link is a separate contribution, and a median over anatomical scale cannot provide it.
Why not just train a fused depth–RGB body network end to end, as Depth-Guided HMR does? End-to-end fusion improves pose in camera coordinates, but it outputs a body, not a world scale. It also learns to trust whichever modality dominates, the collapse documented in Countering Multi-modal Representation Collapse through Rank-targeted Fusion, with no interpretable handle on segmentation reliability. Our D-MAPS-style ablation tests whether its weighting already captures what semantics add.
Proposed Experiments
EMDB is the primary benchmark. Its electromagnetic body sensors and ARKit camera trajectories give metric ground truth for the camera path and global body, with 81 sequences and 0.7% drift over 112-meter walks per EMDB. We will report:
- the ratio of estimated to true camera translation, as a log-scale error;
- world-aligned joint error;
- foot sliding during labeled stance.
These will be computed both with and without dataset intrinsics, to stress metadata dependence.
3DPW checks that camera-frame pose does not regress against the 69.31 mm reference. MEVADA tests pairwise metric distance against Fei et al.'s 78%, using the temporal versions of both methods. VirtualCrowd provides non-planar terrain with ground-truth surfaces. There we will test whether semantic walkability gating prevents contact measurements from being drawn onto curbs and obstacles, and compare against Crowd4D's reported 91.43 PPDS while noting that our method is causal.
The decisive check is the corruption sweep on EMDB. If soft semantic weighting does not flatten the curve of scale error against persistent corruption relative to the temporal median and hard masks, the central claim fails. The second decisive check is the contact ablation on stairs and slopes, which shows whether tying body scale to scene depth through planted feet beats body-only scale.
References
- Single View Physical Distance Estimation using Human Pose
- Depth-Guided Metric-Aware Temporal Consistency for Monocular Video Human Mesh Recovery
- OnlineHMR: Video-based Online World-Grounded Human Mesh Recovery
- Crowd4D: Scene-Aware Monocular 4D Crowd Reconstruction
- VSO: Visual Semantic Odometry
- Contact and Human Dynamics from Monocular Video
- Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation
- Countering Multi-modal Representation Collapse through Rank-targeted Fusion
- EMDB: The Electromagnetic Database of Global 3D Human Pose and Shape in the Wild