Footprints as Static Scene: Coupling Human Contact and Background Reconstruction in Monocular Video

Methods that rebuild a 3D scene from video assume the world holds still. Moving people break that assumption, so current systems either cut the person out with a segmentation mask or give them a free-form deformation. Methods that recover a person's 3D body pose from video have the opposite problem. They need a floor or other surface to keep feet from sliding and sinking, so they either assume a flat infinite plane or require a laser scan of the room. This proposal argues that the two problems share a missing piece. A foot planted on the ground is, for the length of that step, a perfectly static point. It is the one part of the moving person that obeys the static scene assumption. It also lies exactly on the surface the reconstruction is trying to recover.

The idea is to reconstruct the background and the human trajectory together from one monocular video, with a per-foot, per-frame contact variable tying them. During a detected stance, the foot is forced to stay still and to sit on the reconstructed static surface. In the other direction, the planted foot is treated as a measurement of where that surface is. This matters most where the background is hardest to observe: the patch of floor the person hides, low-texture ground, stairs and seats. The claim is that this two-way coupling lets contact-based pose cleanup work on uneven, never-scanned terrain. Current methods cannot do this: they need either a flat-plane model or a pre-scan. The same coupling should also improve background geometry near where people actually walk and sit.

What exists and what is missing

The closest systems each hold one half of the coupling. HUGS jointly fits static background Gaussians and a body-model-anchored human from short monocular videos. Here "Gaussians" means 3D Gaussian Splatting, a scene representation made of many small colored 3D blobs rendered by projection. HUGS consumes precomputed body poses as given, though, so any jitter or foot sliding in them is baked in, and nothing stops the avatar from floating above or sinking into the reconstructed floor. E-Sem3DGS similarly gates each Gaussian as human or scene and uses event-camera flow for temporal coherence, but it does not address foot-ground penetration or sliding. SceneShine composites avatars into reconstructed scenes and reports that ground placement still needs manual alignment. That is direct evidence that nobody geometrically ties the human to the reconstructed surface.

On the pose side, Contact and Human Dynamics from Monocular Video shows what contact reasoning buys. It detects heel and toe contacts and then solves a physics-based trajectory problem, bringing implied ground reaction forces from above 9,000% of body weight to about 140%. It assumes a flat floor at $z=0$, however, and needs 30 to 60 minutes per 2-second clip. SA-HMR lets body vertices attend to points of a pre-scanned scene mesh. On RICH this reduces global joint error to 264.6mm, down from 390.1mm for PROX-style optimization. It needs the scan, though, and it works frame by frame. OnlineHMR masks the person out with SAM2 so its SLAM (simultaneous localization and mapping) camera tracker uses only background pixels. It never reconstructs the background, and it smooths the trajectory with an exponential moving average rather than with contact against a surface. PoseGaussian smooths skeletal pose with a small recurrent module, but that smoothing works in image space and has no scene to press against.

These papers play different roles here. SceneShine and SA-HMR motivate the work: they show that the missing human–scene tie hurts, and that scene geometry helps pose when it is available. Rempe et al. and the densification analysis in RobustSplat constrain the design. The first shows that uniform smoothing cannot fix foot sliding. The second shows that the scene model's rule for adding new Gaussians (densification) must be held back until transient content is explained, or it spawns floating blobs. HUGS and GauHuman supply the method: a skinned human made of Gaussians alongside a static background.

Core Claim

The claim is that stance contacts are brief static anchors, and treating them as such helps both halves of the problem. Let $p_v(t)$ be the world position of foot vertex $v$ at frame $t$. Let $c_v(t)\in[0,1]$ be its soft contact probability. For each stance interval $I$ (a run of frames with high contact), introduce one shared anchor point $a_{v,I}$. Let $d_S(x)$ be the distance from point $x$ to the surface implied by the static background Gaussians, read off their rendered depth and opacity. The coupling term is

$$ L_{\text{contact}} = \sum_{v,I} \sum_{t\in I} c_v(t)\,\lVert p_v(t) - a_{v,I}\rVert^2 \;+\; \lambda \sum_{v,I} d_S(a_{v,I})^2 . $$

The first term makes the planted foot truly static during the step, which removes sliding. The second term makes the anchor lie on the static surface. Its gradient flows two ways: into the body trajectory, which removes floating and penetration, and into the static Gaussians, which pulls surface geometry toward where feet rest. The same idea extends to pelvis-to-seat and hand-to-wall contact. The second term is the new part. Without its gradient into the scene, the method collapses to "fit a scene, then fix the pose against it."

Research Program

We would build one offline, windowed optimizer over three sets of variables. The first is static background Gaussians, initialized from depth and SLAM points. The second is a human modeled with the standard SMPL body mesh and Gaussians skinned to it as in GauHuman, starting from off-the-shelf world-grounded poses. The third is camera poses. Photometric loss applies to both branches. The human's rendered silhouette explains its own pixels, so the background is not asked to absorb the person. Following RobustSplat, background densification is delayed until the human branch is well fitted. We would also forbid densification inside the human's current silhouette, which targets the floaters that jittery poses would otherwise spawn along the trajectory.

Contact probabilities come from a learned detector, as in Rempe et al., and are then refined inside the optimization. A contact persists only if the anchor can reach a surface without violating the photometric fit. This gives a principled way to reject false contacts, and false contacts matter given SA-HMR's modest contact precision of 0.57 and recall of 0.53.

We would also analyze a scale question. Monocular reconstruction has an unknown scale, and body height is a known quantity. Stance anchors therefore tie scene scale to human scale at many points over time. We will measure whether this reduces camera-trajectory scale drift relative to a masked-SLAM pipeline.

Baselines

The naive reference is per-frame world-grounded pose from the same initializer, followed by a standard temporal filter on joints and root, with the background reconstructed separately by masked static Gaussian Splatting. It uses the same initial poses and the same number of optimization steps for the background as our method, so any gain comes from coupling, not from extra compute.

The strongest competitor on non-flat, unscanned terrain is a decoupled "reconstruct, then fit" pipeline. It masks the person with SAM2, reconstructs the background with RobustSplat's delayed-growth schedule, extracts a surface, and then runs our contact optimization against that frozen surface. It is identical to our method except that no gradient flows from contacts into the scene. It is also the fairest ablation of the key ingredient, since it shares every component, tuning budget and contact detector. Beating it means lower foot sliding and penetration, and lower surface error in contact regions, with the gap concentrated on stairs, seats and occluded floor.

The oracle-geometry reference is SA-HMR given the ground-truth RICH scan. It reports G-MPJPE of 264.6mm on RICH (SA-HMR). Our method never sees the scan. Matching or approaching that number without the scan would count as success, and beating it would show that temporal contact reasoning adds what per-frame attention lacks.

The fourth baseline is a flat-ground contact model. It fits a single plane from metric depth, imposes stance zero-velocity and plane contact in the manner of Rempe et al. but using gradient descent, and is tuned on the same budget.

For world-coordinate pose on EMDB, OnlineHMR is the online reference. The file reports only that it outperforms TRACE and Human3R and matches offline chunk-based models; no reported result found on Lacuna for its WA-MPJPE value. We will run it ourselves on identical sequences.

Why not just…?

Why not just smooth the poses with a Kalman or One-Euro filter afterward? Uniform filters average stance frames with swing frames. That averaging produces the foot sliding we aim to remove, and Rempe et al. show that kinematic smoothing leaves implied forces far above physical values. The naive baseline measures this directly through the foot-sliding distance during labeled contact.

Why not just fit a ground plane and use flat-floor contact? This is a real threat. On EMDB's mostly flat walking sequences, we expect it to roughly tie our method on pose metrics, and we will report that honestly rather than argue around it. The proposal's claim is therefore restricted to terrain that a plane cannot represent. The flat-ground baseline is evaluated separately on the RICH and PROX sequences that contain stairs, slopes, sofas and chairs. If it matches us there too, the scene-coupling claim fails.

Why not just mask the human, reconstruct the background, and then run scene-aware pose fitting against it? This is the decoupled baseline above. The argument against it is specific: the floor under and around the person is exactly where the masked reconstruction has the least evidence. The person occludes it, and SAM2 masks bleed at the feet. So the surface that pose fitting depends on is weakest precisely where it is needed. Our contact-region geometry metric tests whether the scene-directed gradient actually repairs that surface. If it does not, the decoupled pipeline is sufficient and the paper's second claim is false.

Why not just use SA-HMR with a pre-scan? Pre-scans do not exist for in-the-wild video such as NeuMan or EMDB. SA-HMR is also per-frame, so it offers no temporal stance constraint. We include it as an oracle, not as a practical alternative.

Proposed Experiments

The main testbed is monocular video from RICH and PROX. The scanned meshes serve only as ground truth for geometry and contact, never as input. We report G-MPJPE, contact precision and recall, penetration depth, and foot-sliding distance during stance. We also report Chamfer distance between the reconstructed static surface and the scan, computed separately in a band around ground-truth contact regions and over the whole scene. The decisive check is the contact-band Chamfer distance against the decoupled baseline. The coupling claim needs a clear improvement there, with little change elsewhere.

EMDB-2 tests world-grounded pose on long outdoor trajectories with WA-MPJPE and trajectory drift. These sequences also host the flat-ground comparison, where we expect parity. NeuMan tests whether coupling costs rendering quality, measured by novel-view PSNR, SSIM and LPIPS against HUGS on identical frames; the file gives no reported HUGS or SceneShine number on NeuMan, so we will rerun both. We will also count background Gaussians that fall inside the human's swept volume, to test whether silhouette-gated densification removes trajectory floaters.

A final stress test perturbs the initial poses with synthetic depth-flip noise. It asks whether scene contact corrects the depth ambiguity that per-frame estimators suffer from, or merely hides it. The dimensions that decide success are three. Pose on non-flat terrain must beat the flat-ground baseline. Contact-region geometry must beat the decoupled pipeline. Rendering quality must hold on NeuMan.