Footprints as Ground Truth: Joint Recovery of Human Motion and Unscanned Terrain from a Moving Camera

Systems that reconstruct a walking person in 3D from a handheld video usually assume the floor is one flat, horizontal plane. The assumption makes the problem tractable, but it breaks on stairs, ramps and curbs: the reconstructed person floats above the surface or sinks into it. The methods that handle uneven ground well need a laser scan or pre-built height map of the environment. A handheld video of a person already contains two partial views of the ground, and they fail in complementary places. The background pixels, reconstructed across frames, show the terrain wherever the person is not standing. The person's footfalls touch the terrain exactly where the background view is worst: under the body, hidden by the legs, often on textureless floor. We propose to recover both together. A terrain surface starts from a background reconstruction that treats the person as a moving distractor. Detected foot contacts then pin that surface down at metric scale. The resulting terrain in turn corrects the foot and root estimates at the moments when the 2D keypoint detector is least reliable, which is at contact.

The central claim is that this two-way coupling beats the obvious pipeline: reconstruct the background first, then run a terrain-aware human tracker on the result. It should win most clearly on terrain height directly under footfalls, on metric scale, and on frames where the contacting foot is occluded. If the coupling does not beat that pipeline, the proposal fails, and the experiments are designed so that this outcome would be visible.

What is actually missing

Three bodies of work each solve part of the problem. The strongest non-flat tracker, PhysDynPose, combines a root-relative body estimator and visual SLAM (simultaneous localization and mapping, which estimates camera motion from video) with a physics simulator and an explicit height map. On non-flat MoviCam scenes it lowers root translation error from 2.19% for WHAM to 1.16%. However, the height map comes from a pre-scanned mesh. WHAM handles stairs without a scan, but only implicitly: it uses learned contact probabilities and never reconstructs the surface. Transformer Inertial Poser shows that terrain height can be inferred from foot contacts alone, but it works from body-worn motion sensors and never sees terrain the person did not step on.

Flat-ground methods constrain the design. GraMMaR estimates the ground as a single plane with 0.999 cosine similarity on AMASS. PPR uses gravity as a metric ruler but explicitly assumes one flat plane. Both indicate that contact-derived ground estimates are accurate when the ground model has the right form. The limitation is the planar form, not the contact signal.

On the reconstruction side, RobustSplat removes moving distractors from a Gaussian-splat scene. It delays densification, the step where the scene representation adds new primitives wherever rendering error is high, so that moving objects do not leave floating artifacts. It then discards the distractors entirely. Split4D shows that resolving dynamic identity in 3D rather than through 2D video trackers raises mIoU from 0.592 to 0.893. Panoptic Lifting shows that 3D consensus cleans up noisy 2D labels, but only in static scenes.

Across these, the person is either the target or the noise, never a measurement of the scene. BSTRO supplies the key motivating observation, which it calls the occlusion paradox: contact regions are self-occluded exactly when contact matters. Contact must therefore be inferred from context, and a reconstructed terrain is exactly that context.

Core Claim

A terrain field that is reconstructed photometrically from background pixels and anchored at footfalls will do three things better than either source alone, and better than a one-way pipeline built from the same parts. It will give lower terrain error under the walked path. It will give correct metric scale. It will give lower root drift and foot sliding than a pipeline that first builds the terrain and then refines the human against it.

The dynamic-scene contribution is specific. The mask that separates the moving person from the background is rendered from the current 3D body estimate, not taken from a 2D segmenter. This avoids the mask-propagation failures documented for Split4D's baselines. It also means that improvements to the body estimate improve the background reconstruction.

Research Program

The scene is represented as a static Gaussian-splat background plus a height field $h(x,y)$, a function giving terrain height at each horizontal location. The height field is extracted from the splats and refined as a free variable. The person is represented as an SMPL body trajectory, initialized from WHAM. Camera poses come from DROID-SLAM and are re-estimated with pixels inside the rendered body silhouette down-weighted.

Optimization alternates between two blocks. In the background block, a RobustSplat-style reconstruction runs with delayed densification. Its transient mask is the union of RobustSplat's bootstrapped mask and the projected SMPL silhouette, dilated by an uncertainty margin.

In the human block, the body trajectory and the height field are updated under the following objective:

$$ \mathcal{L} = \mathcal{L}_{2D} + \lambda_c \sum_{t,j} c_{t,j}\left[(z_{t,j} - h(x_{t,j},y_{t,j}))^2 + \|v_{t,j}\|^2\right] + \lambda_p \sum_{t,j} \max(0, h(x_{t,j},y_{t,j}) - z_{t,j})^2 + \lambda_h \mathcal{L}_{photo}(h) $$

The terms are defined as follows. $\mathcal{L}_{2D}$ is the confidence-weighted 2D keypoint reprojection loss. $c_{t,j}$ is the predicted probability that foot joint $j$ is in contact with the ground at frame $t$. $(x_{t,j},y_{t,j},z_{t,j})$ is that joint's world position and $v_{t,j}$ its velocity. The $\lambda_p$ term penalizes the body penetrating the terrain. $\mathcal{L}_{photo}(h)$ ties the height field to the splat geometry, and is weighted by how often each terrain cell was observed unoccluded.

Two design choices carry the idea. First, under a footfall the height field trusts contacts, and elsewhere it trusts photometry. The photometric weight falls as a cell's unoccluded visibility falls, so footfalls dominate exactly where background observations are missing. Second, when the contacting foot's 2D confidence is low, the keypoint term is down-weighted and the terrain term dominates. That is the intended correction for the occlusion paradox.

Metric scale is fixed jointly: SMPL body dimensions set the scale of the camera trajectory and the splats together. Full physics simulation of the PhysDynPose kind is an optional final stage, not the core. PPR's reported 18 hours per sequence is a warning about cost, so the core stays kinematic, and runtime is reported for every configuration.

Baselines

The naive reference is WHAM run as released. It represents the no-explicit-terrain, flat-plus-learned-contact approach, and on MoviCam non-flat terrain it reports a root translation error of 2.19% (WHAM as reported by PhysDynPose). 4DHumans and GLAMR give weaker kinematic references, with foot sliding of 13.99 mm and 15.01 mm and GLAMR's root translation error at 2.92%. Beating WHAM on root translation error by a clear margin, with our foot sliding well below the 13.99 mm 4DHumans figure, is the minimum bar. The file reports no WHAM foot-sliding number.

The strongest existing method is PhysDynPose given the pre-scanned MoviCam mesh, with a reported 1.16% root translation error and 3.22 mm foot sliding. It uses information our method does not have, so it acts as an oracle ceiling rather than a competitor to beat. Success means closing most of the gap from WHAM's 2.19% to 1.16% without any scan.

The fair competitor is the cascade. This is PhysDynPose's refinement, and separately our own human-block losses, run on a height map extracted from RobustSplat applied to the same video, with DROID-SLAM cameras and metric scale set from the WHAM body. All methods use the same frames, the same WHAM initialization, and the same 2D detector. Iteration budgets and wall-clock time are matched, and hyperparameters are tuned on a held-out MoviCam split for every method.

Two ablations isolate the key ingredient. The first freezes the height field after the background block, which gives one-way coupling. The second drops photometry entirely and fits terrain from contacts alone, in the style of Transformer Inertial Poser. A third ablation replaces the SMPL-rendered transient mask with a 2D video segmentation mask. It tests whether the body-driven mask matters.

Why not just…?

Why not just run RobustSplat, extract a mesh, and feed it to PhysDynPose? This is the serious threat, and we build it as a baseline instead of arguing against it. Its weakness should be concentrated in specific places: terrain under and behind the walking person, which the handheld camera sees mostly occluded by the legs; textureless floors; and scale, since monocular reconstruction is only defined up to scale. We will report terrain height error separately inside the walked corridor and outside it. If the cascade matches the joint method inside the corridor, the coupling adds nothing. We would then report that result and fall back to the simpler pipeline.

Why not just use WHAM, which already handles stairs? WHAM's contact-aware refinement never produces a surface. PhysDynPose's gap over WHAM on MoviCam, 1.16% against 2.19% root translation error, is evidence that explicit terrain helps. Our question is whether that terrain can be obtained without a scan.

Why not infer terrain from footfalls alone? Transformer Inertial Poser does this with inertial sensors, and our contact-only ablation reproduces the idea visually. It cannot place terrain the person has not yet touched. Its errors should therefore appear at the first step onto each new surface, which is where penetration and float events occur. We will measure errors on first-contact frames specifically.

Why not regress body and scene directly from pixels? The file's evidence on fine foot articulation points the other way. FootMR reports a 30.6% ankle-angle error reduction from lifting with knee context rather than relying on direct video regression. No regression model in the file outputs metric terrain.

Proposed Experiments

The main benchmark is MoviCam. It provides non-flat scene meshes, global human pose, and moving-camera video, which lets us score root translation error, foot sliding and penetration, and terrain height error against the scanned mesh. RICH provides laser-scanned scenes and dense contact labels. There we compare contact F1 against BSTRO's reported 0.708, and HuMoR's 0.314, using the terrain field to classify contact. EMDB tests generalization of trajectory quality. The file gives no per-benchmark numbers for EMDB, so we will rerun WHAM on it ourselves.

Three checks decide the outcome. The first compares terrain error inside the walked corridor for the joint method against the cascade. The second compares root translation error and foot sliding against the cascade on frames where the contacting foot's 2D keypoint confidence falls below a threshold, which directly tests the occlusion-paradox claim. The third reports metric scale error of the reconstructed camera trajectory. If the joint method is only marginally better than the cascade on all three, the conclusion is that pipelines suffice once background reconstruction is robust to moving people, and that finding is worth reporting as well.