Footsteps as Landmarks: Contact-Anchored Online Reconstruction of Humans and Scenes

When a person walks through a phone video, the camera sees two things that are hard to reconstruct together: a static room and a moving body. Current systems that build both at once, using 3D Gaussian splatting (a scene representation made of many small colored ellipsoids that can be drawn very fast on a GPU), treat the person and the room as separate objects that meet only when one hides the other. This leaves two persistent errors. Because a single camera cannot measure depth directly, the person's distance from the lens drifts. Because nothing ties the feet to the floor, the reconstructed body floats above the ground, sinks into it, or slides along it.

The opportunity is that a walking person is not always moving. Every step holds a foot still on the floor for a few hundred milliseconds. During that interval the foot is effectively part of the static scene. The camera moves around it, so it can be triangulated from several viewpoints, and it touches the reconstructed floor at a definite point. We propose to detect these stance intervals online and turn each planted foot into a temporary static landmark. The landmark is shared by three components: the camera tracker, the articulated body model, and the static scene map. Each footstep then supplies an exact, physically grounded constraint that fixes how far the body is from the camera, how large the scene is, and where the floor surface lies, all without adding meaningful cost to a system that renders in real time.

Core Claim

Treating stance-phase body parts as intermittent static landmarks in the bundle adjustment of an online human-and-scene reconstruction system will reduce world-frame human pose error and camera trajectory error. It will also remove foot sliding and ground penetration, and it will keep doing so on stairs, slopes, and cluttered floors where a global ground-plane assumption fails. Bundle adjustment here means the joint optimization of camera poses and 3D points so that they agree with the image observations. The new ingredient is not a contact penalty in itself, since kinematic contact losses already exist. It is the use of contact as multi-frame static evidence inside camera tracking. That evidence converts the depth ambiguity of a moving body into a triangulation problem of the kind a static scene already gets.

Concretely, let $\mathbf{p}_f(t)$ be the world-space position of a foot keypoint on the articulated avatar at frame $t$, let $c_f(t)\in[0,1]$ be its estimated contact probability, and let $\mathbf{l}_k$ be a landmark created for the $k$-th detected stance interval $\mathcal{I}_k$. We add

$$ \mathcal{L}_{\text{contact}} = \sum_k \sum_{t\in\mathcal{I}_k} c_f(t)\,\rho\!\left(\lVert \mathbf{p}_f(t)-\mathbf{l}_k\rVert\right) + \lambda \sum_k \rho\!\left(d_{\mathcal{S}}(\mathbf{l}_k)\right), $$

where $\rho$ is a robust (Huber-style) penalty that limits the damage from false contacts, $d_{\mathcal{S}}(\mathbf{l}_k)$ is the signed distance from the landmark to the surface rendered from the static Gaussians, and $\lambda$ weights surface attachment. The first term says the foot does not move during stance. The second term says it rests on whatever surface is actually there, not on an assumed plane. The landmark $\mathbf{l}_k$ also enters the camera-tracking residuals, through the reprojection of the foot keypoint, in the same way as any static map point.

Positioning Against Prior Work

The base system comes from ODHSR. It keeps static scene Gaussians alongside avatar Gaussians posed by linear blend skinning, which deforms a template body by blending bone transforms driven by the SMPL parametric body model, plus a neural residual. It runs separate tracking and mapping threads online and renders at about 85 FPS. It already uses the body's known size as a scale cue. Its authors also report the failure we target: feet that penetrate or float, because the system has no contact constraints. We reuse its architecture and change what flows between the avatar and the map.

Two papers motivate the contact signal and also limit how we use it. Reducing Footskate detects stance from 2D keypoint sequences with a temporal convolutional network (contact mAP 0.9292 on Human3.6M) and applies a zero-velocity loss to the kinematic trajectory. That loss acts only on the body, after the fact, and never touches the camera or the scene. From-Ground-To-Objects forces the base of moving objects to the depth of the road plane, which corrects monocular depth for cars and pedestrians. It assumes one flat ground and works on depth maps rather than a renderable 3D model. We take the stance detector from the first paper and the insight from the second that contact resolves depth. We replace the global plane with the locally rendered static surface, and we replace a post-hoc trajectory fix with landmarks inside tracking.

Two further lines of work act as constraints on the design. WildGS-SLAM shows that down-weighting moving pixels through learned uncertainty protects the static map extremely well (0.46 cm trajectory error on Wild-SLAM MoCap). It discards the person entirely, and with them the contact cues we exploit. MoDGS shows that per-frame foundation depth is scale-inconsistent over time, which is why it supervises only depth order. This warns us that leaning harder on the depth network ODHSR already uses (Depth Anything v2) will not fix the drift. USplat4D models depth uncertainty as larger along the viewing ray, and we borrow that idea for the foot keypoint residual. Its dense graph and the control-point skinning of 4D3R are offline or slower and do not couple the subject to the scene.

Research Program

We will build on ODHSR's two-thread design. In the tracking thread, a contact module combines a temporal keypoint detector, trained as in Reducing Footskate, with a geometric check: the foot is a contact candidate only if its rendered depth lies within a tolerance of the static Gaussians' depth along the same ray. When a stance interval opens, we create a landmark at the foot's current estimate. While the interval is open, the landmark's reprojection and the avatar-to-landmark residual join the per-frame camera and joint-angle optimization. When the interval closes, the landmark is kept for a window and then marginalized. The mapping thread uses landmark positions to pull nearby floor Gaussians toward the surface. Stairs and slopes are handled because each landmark attaches to its own local surface patch.

The analysis will separate three effects: depth along the ray for the person, global scale for the camera, and surface quality of the floor. We will also measure how often false contacts occur and what each one costs. All of this must preserve real-time operation. The extra residuals number only a few per frame, so we expect negligible tracking overhead, and we will measure it rather than assume it.

Baselines

The naive reference point is ODHSR as published, run with its released settings on the same sequences. It reports on EMDB a whole-image PSNR of 23.79, a camera trajectory error of 8.4 cm, and a world-aligned joint error (WA-MPJPE) of 175 mm. This is also the strongest existing method on the question, because it is the only system in the file that jointly reconstructs an articulated person and the static scene from monocular video online. We will rerun it ourselves on identical hardware and use the reproduced numbers as the comparison, with the published figures as a sanity check. The proposal beats it if WA-MPJPE and trajectory error both fall clearly beyond run-to-run variation across seeds, foot sliding and penetration fall substantially, and PSNR and rendering rate stay within noise.

The second baseline isolates whether contact needs to live inside tracking. It is ODHSR followed by the zero-velocity trajectory optimization of Reducing Footskate, using the same contact detector we use. The third isolates whether the local surface matters. It adds to ODHSR a global ground-plane height penalty in the spirit of From-Ground-To-Objects, applied inside tracking, with the plane fitted to the static Gaussians. The fourth is a decoupled pipeline: the person is masked with WildGS-SLAM-style uncertainty for static SLAM, and an avatar is fitted separately with GauHuman. This tests the claim that the subject should help the map rather than be erased from it.

Our own ablations remove one ingredient at a time: the surface-attachment term, the landmark reprojection in camera tracking, and the geometric gate on contact detection. All variants share the same depth and keypoint networks, keyframe windows, and iteration budgets. We will tune hyperparameters on held-out sequences under the same budget for every method.

Why not just…?

Why not just apply a footskate cleanup to ODHSR's output? A post-hoc zero-velocity fix moves the feet but leaves the camera trajectory, the scale, and the floor Gaussians untouched. It can therefore remove sliding while the whole body stays at the wrong distance from the camera. The second baseline tests this directly. If it matches our method on WA-MPJPE and camera trajectory error, the coupling we propose is unnecessary and we will report that. We expect it to improve the sliding metrics only.

Why not just penalize foot height above a fitted ground plane? This is the real threat. On flat floors, intersecting the viewing ray through the foot with a known plane already fixes the foot's depth, which is exactly why From-Ground-To-Objects works. Much of EMDB and NeuMan may be close to flat, so the third baseline could match us there. We will therefore stratify results by terrain. The claim stands only if the full method wins clearly on stairs, slopes, curbs, and furniture contact, and also matches the plane baseline on flat ground through its stance-landmark term, which removes in-plane sliding that a height penalty cannot. If the plane baseline wins or ties everywhere, the honest conclusion is that local surface attachment is not worth its complexity.

Why not just use a stronger metric depth network? ODHSR already starts from Depth Anything v2, and its residual errors on EMDB come after that initialization. MoDGS shows that single-frame depth fluctuates in scale across frames, so tighter supervision on those values distorts geometry. We will include a variant that adds a dense metric depth loss to ODHSR to check this in our setting.

Proposed Experiments

The main evaluation uses EMDB. It gives in-the-wild global human motion and camera trajectories from body-worn electromagnetic sensors, which the EMDB authors validate at 2.3 cm positional error, so it can resolve improvements well below ODHSR's 175 mm. We will report WA-MPJPE, camera trajectory error after scale alignment, and scale error without alignment. Two foot metrics complete the set: average foot displacement during ground-truth stance, and penetration depth relative to the reconstructed floor surface. The file reports no result on Lacuna for either foot metric on EMDB for any baseline, so we will compute them for every method. EMDB's ground-truth foot heights over time also let us label sequences as flat or uneven terrain without manual annotation.

For appearance and generality we will use the NeuMan phone videos, reporting PSNR on held-out frames against the reproduced ODHSR. For reference, NeuMan itself reports 23.98 PSNR on Seattle. Rendering rate and per-frame tracking time are measured on a single GPU for every method.

Two checks decide the outcome. The first is the terrain split against the plane baseline described above. The second is a stress test of false contacts: we inject spurious stance detections at controlled rates and measure how quickly trajectory error degrades with and without the geometric gate and the robust penalty. A method whose gains disappear at realistic detector error rates would not be worth adopting. If both checks pass, a natural follow-up is to extend stance landmarks to other articulated subjects such as animals, whose skeletons can be recovered automatically with RigGS.