When a single moving phone camera films a scene, geometry can be recovered only up to an unknown scale factor. This is the monocular scale ambiguity. Recent systems fix the factor by using a person in the video as a ruler, because bodies have roughly known sizes and walking speeds. These systems all assume that everything except the person is one rigid, motionless world. That assumption hides a basic fact. When other things move, such as a cart, an escalator, a train car, or a door, each independently moving rigid object is reconstructed with its own separate unknown scale. A single image sequence cannot tell a small nearby cart from a large distant one moving faster. So a dynamic scene has one unknown scale per moving body, and existing human-as-ruler methods fix only one of them.
This note proposes a way to fix the others. A person can pass metric scale to whatever they physically touch. Feet planted on an escalator step, or hands on a moving cart, tie that object's surface to a body of known size. The person's walking speed relative to the surface they stand on also reveals how fast that surface itself is moving. We propose a joint optimization over camera, people, and several rigid bodies. Its key change is that "this foot is not sliding" is measured relative to whatever the foot is standing on, not relative to the world. The output is metric geometry and motion for every object a person interacts with. It also flags objects that no person touches, whose scale is left to weaker cues.
Core Claim
The claim has two parts, and they can be tested separately. The first part is about contact. Moving supports are a common failure of contact-based joint optimization: escalators, moving walkways, vehicles, boats, and pushed carts. In these cases, rewriting the "foot does not slide" constraint in the support body's own coordinate frame removes the errors that a world-frame constraint introduces. Meanwhile, results on static-scene benchmarks stay unchanged, because a static world is just the special case where the support's motion is zero.
The second part is about scale. Contact plus a walking-speed estimate pins down the metric scale of a touched moving object more accurately than the obvious alternative, which is to read that object's scale from a per-frame metric depth network.
If the second part fails, the first may still hold. The experiments are designed so we learn which is which.
Where This Sits
Three systems supply the method we build on.
JOSH jointly refines focal length, camera poses, a dense scene point cloud, and SMPL bodies (SMPL is a standard parametric human body model). It uses two contact losses: one pulling foot contacts onto scene points ($L_{c1}$), and one forcing contacts to stay still in world coordinates ($L_{c2}$). This cut foot floating on SLOPER4D from 9.0% to 2.9%.
HSfM uses body proportions to put uncalibrated scene pointmaps into meters. It shows that human keypoints improve camera rotation by over 20%.
WHAC predicts a person's absolute speed in meters per second from body motion, using a module it calls the MotionVelocimeter. It uses that speed to scale the camera trajectory, and it explicitly discusses the treadmill and skateboard cases, where body motion and world motion disagree.
We reuse JOSH's pointmap-plus-SMPL optimization, HSfM's body-as-ruler scale term, and WHAC's speed prediction. The difference is the same for all three: they treat the non-human scene as one rigid frame. JOSH's $L_{c2}$ in particular actively pushes the solution wrong on a moving support. It forces a person riding an escalator to be stationary in the world, and that error has to be absorbed by the camera path or the scene scale.
Several works motivate the gap without filling it. ODHSR models one person as dynamic Gaussians (small 3D blobs that are rendered into images) over a static Gaussian scene, and uses the body to stop the camera's scale from drifting. Its own limitations note that any non-human motion violates its static-scene part. Crowd4D anchors many people to a non-flat ground surface, but that surface must be motionless and the camera fixed. ArtHOI reconstructs a moving articulated object and then fits hands to it. Its metric scale comes from asset-library templates rather than from the person, and its camera is fixed.
Two lines of work constrain the design. RobustSplat shows that moving content corrupts static reconstructions, and its answer is to mask it out. Monocular Dynamic Gaussian Splatting: Fast, Brittle, and Scene Complexity Rules shows that generic dynamic Gaussian methods are brittle on casual phone video. Both push us away from a free-form deformation field and toward a small number of explicit rigid bodies.
USplat4D makes the related point that monocular error is concentrated along the viewing ray. Per-body scale is the rigid-body version of that same ray-direction ambiguity.
Research Program
The scene is modeled as a person (or several people) plus a static background, body 0, plus $K$ moving rigid bodies. Each body $b$ has a pointmap in its own frame, a pose $T_b(t)$ over time, and a scale $s_b$. The background scale $s_0$ is fixed by human size and ground contact, exactly as in JOSH and HSfM. Initialization follows JOSH. Per-frame pointmaps and SMPL fits come from the same off-the-shelf estimators, and point tracks are grouped into rigid bodies by shared 3D motion. The grouping follows the idea behind Split4D, which lets object identity emerge from motion coherence, but here it is applied to point tracks rather than Gaussians.
Each detected contact $c$ is assigned to a support body $b_c(t)$. This assignment is a hidden variable, re-estimated by nearest surface after each optimization round. The world-frame stationarity loss is then replaced by a support-frame one:
$$L_{\text{stay}} = \sum_{c,t} \left\| T_{b_c}(t)^{-1}\, x_c(t) - T_{b_c}(t+1)^{-1}\, x_c(t+1) \right\|^2 .$$
Here $x_c(t)$ is the world position of the contact vertex on the SMPL mesh at frame $t$. When $b_c = 0$ and the background does not move, this reduces exactly to JOSH's $L_{c2}$. The surface-alignment loss $L_{c1}$ is applied in the support frame too. Because the SMPL mesh is metric, this alignment is what fixes $s_{b_c}$.
The gait term couples speeds. The person's world velocity must equal the support's velocity at the contact point plus the predicted speed relative to the support:
$$v_{\text{world}}(t) \approx v_{b_c}(t) + \hat{v}_{\text{gait}}(t).$$
Here $\hat{v}_{\text{gait}}$ is a WHAC-style speed prediction from canonical pose alone. For a person standing still on an escalator, this term gives the escalator's metric speed directly, even before contact geometry is accurate.
Bodies that no person ever touches keep a scale prior from per-frame metric depth. They are reported as scale-uncertain, so the system does not claim more than it knows.
What we build is the optimizer with these three terms, the support assignment step, and an evaluation set where moving supports have known metric geometry. What we analyze is when each channel carries the scale: size, contact, or speed.
Baselines
The naive reference is a single global scale factor set by assuming a 1.70 m person, applied to the per-frame pointmaps with no further optimization. This is the Rother et al.-style ruler, and Lin warns that it confounds a person's height with their distance. It sets the floor for every metric. Beating it should be easy; failing to beat it by a wide margin would signal a bug.
The strongest existing method is JOSH, run from its released configuration on identical initial estimates, identical inputs, and the same iteration budget. We will not retune it in a way that disadvantages it. On static data, the reported numbers to match are a foot floating rate of 2.9% and the roughly 46% WA-MPJPE reduction on SLOPER4D (JOSH). WA-MPJPE is joint position error after aligning the whole trajectory to ground truth. Beating JOSH means two things. On static benchmarks, we must be within noise of its numbers. On moving-support sequences, we must clearly reduce camera trajectory error, WA-MPJPE, and foot skating measured in the support's frame.
For camera and person accuracy on EMDB2, we compare against WHAC, which reports a WA-MPJPE of 142.2 mm, and ODHSR, which reports a WA-MPJPE of 175 mm and a camera trajectory error of 8.4 cm on EMDB. Both are run on the same sequences.
A masking baseline matters most to a skeptic. It segments and removes moving regions RobustSplat-style, then runs JOSH on the remainder.
The ablations isolate the key ingredient one at a time. The first restores world-frame $L_{c2}$ while keeping the multi-body scene. The second drops the gait term. The third drops contact-based scale transfer, so moving bodies take their scale only from metric depth.
Why not just…?
Why not just mask moving objects and run JOSH on the static remainder? For the person and the camera on ordinary sidewalks, this is a real threat. It may match us, and the static-benchmark comparison will show whether it does. It fails in two specific situations. When the support itself moves, masking the escalator or train floor deletes the very contact JOSH needs. The person's world motion is then indistinguishable from walking, which is WHAC's treadmill confusion in reverse. Masking also returns no metric geometry for the moving objects at all. If masking ties us on moving-support sequences, the contribution reduces to "segment, then fit," and we would report exactly that.
Why not just read each moving object's scale from a per-frame metric depth network such as Metric3D v2? This is the strongest alternative for the scale claim. It needs no person, and it gives meters directly by normalizing every camera to a virtual reference camera. We do not know its per-object scale error on moving objects in casual video. No reported result was found on Lacuna for per-object metric scale error of Metric3D v2 on dynamic scenes. The evidence against relying on it alone is indirect: SHARE and ODHSR describe frame-to-frame scale flicker in per-frame depth. Our experiment settles it directly. If time-smoothed Metric3D v2 brings per-object scale within about 5% of ground truth on our sequences, we drop the contact-scale claim and keep only the support-frame contact result.
Why not just fit a generic dynamic Gaussian model and rescale it afterwards with the person? A single global rescale cannot fix the separate per-body scales: rescaling the whole scene leaves a mis-scaled cart mis-scaled relative to the floor. The benchmark in Monocular Dynamic Gaussian Splatting: Fast, Brittle, and Scene Complexity Rules also shows that these models are brittle on phone captures. We will still run USplat4D, rescaled by the person, on the synthetic set as a reference for geometry quality.
Proposed Experiments
The first experiment is a non-regression test on static scenes: EMDB2, SLOPER4D, and RICH. It uses the metrics those papers already report, namely WA-MPJPE, foot floating rate, and camera trajectory error. The expectation is a tie with JOSH, because the support-frame loss reduces to JOSH's loss in a static world. A loss of more than a few percent would mean the support assignment is introducing errors.
The second experiment needs new data, because no benchmark in the file contains moving supports. It has two parts.
The first part is a controlled synthetic set in the spirit of SlidingCube. SMPL people stand on, walk along, or push platforms, walkways, and carts. Platform speed, camera baseline, and contact visibility each vary independently, and every body has exact metric geometry.
The second part is a small real set captured with a LiDAR-equipped phone. It covers escalators, moving walkways, train interiors, and pushed carts with hand-measured dimensions. It is necessarily small and is used for validation, not training.
On both parts we measure four things. Per-body scale error is $|\log(s_b/s_b^\ast)|$, the log ratio of estimated to true scale. Object shape error is Chamfer distance in meters. Support speed error is in meters per second. Person WA-MPJPE and foot skating are measured in the support's frame.
The decisive checks are three. The world-frame-$L_{c2}$ ablation should degrade sharply as platform speed rises. Its error should grow roughly in proportion to the platform speed that the camera path is forced to absorb, while our method stays flat. Next, scale from contact transfer should beat smoothed Metric3D v2 on touched bodies, and the gap should shrink when contacts are hidden. Finally, on the escalator subset, the gait term alone, with contact geometry frozen, should recover belt speed. That result would show that walking speed carries metric information about the support and not only about the camera.
References
- Joint Optimization for 4D Human-Scene Reconstruction in the Wild
- Reconstructing People, Places, and Cameras
- WHAC: World-Grounded Humans and Cameras
- ODHSR: Online Dense 3D Reconstruction of Humans and Scenes from Monocular Videos
- Crowd4D: Scene-Aware Monocular 4D Crowd Reconstruction
- ArtHOI: Articulated Human-Object Interaction Synthesis by 4D Reconstruction from Video Priors
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- Monocular Dynamic Gaussian Splatting: Fast, Brittle, and Scene Complexity Rules
- Uncertainty Matters in Dynamic Gaussian Splatting for Monocular 4D Reconstruction
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- What Can Casual Walkers Tell Us About A 3D Scene?
- HDNet: Human Depth Estimation for Multi-person Camera-Space Localization
- Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation
- SHARE: Scene-Human Aligned Reconstruction