The Ground Agrees: Learning Foot Contact from Unlabeled Real Video

When a foot is planted, it stops moving relative to the floor beneath it. In a video, this is visible without any 3D labels. A point on the planted heel and a point on the floor next to it move together across the image, however the camera pans or shakes, because both are attached to the same static world for that instant. We propose to use this fact as free supervision. We track points on the feet and on nearby ground in ordinary real videos, label a foot region as "in contact" when its track matches the adjacent ground track, and use those labels for two jobs. First, we train a per-frame contact detector directly on real footage. Second, we add a cheap image-space anti-sliding term to monocular 3D human motion recovery.

The opportunity sits where three problems meet. Real video of people almost never comes with ground-truth contact or 3D foot trajectories. Monocular pose estimators therefore produce feet that glide or "skate" across the floor. The usual workaround, training contact or pose models on synthetic humans, runs into the gap between synthetic and real imagery. The literature has left a contradiction unresolved. Synthetic-trained models must discard RGB appearance to transfer, yet the cues that reveal contact, such as shadows, shoe–floor darkening and clothing folds, are appearance cues. Supervising on real video with a geometric signal removes that tradeoff: the detector can look at real pixels because it never learns from rendered ones.

Core Claim

Agreement between foot and ground motion in point tracks is a supervision signal that works on any real video. It is independent of camera motion, needs no depth, and does not depend on appearance. A contact detector that looks at RGB and is trained only on these pseudo-labels from unlabeled real video should beat two references: detectors trained on synthetic contact labels, and fast pose pipelines that infer contact by thresholding their own estimated 3D foot velocity. Used as a differentiable loss, the same signal should remove most foot sliding at feed-forward cost rather than physics-optimization cost.

Concretely, let $\mathbf{p}^f_t$ be the 2D image position of a tracked point on a foot region (heel or toe) at frame $t$, and $\mathbf{p}^g_t$ the position of a tracked ground point within a small image neighborhood of it. Let $s_t$ be the foot's image length, used to normalize for scale. The relative residual and soft pseudo-label are

$$r_t = \frac{\|(\mathbf{p}^f_{t+1}-\mathbf{p}^f_t)-(\mathbf{p}^g_{t+1}-\mathbf{p}^g_t)\|}{s_t}, \qquad \hat c_t = v_t \exp(-r_t^2/\sigma^2),$$

where $v_t\in[0,1]$ is the tracker's joint visibility confidence for both points and $\sigma$ is a single threshold calibrated on synthetic data where true contact is known. Heel and toe are treated separately, so heel-strike, flat-foot and toe-off appear as distinct phases rather than one binary state. This also addresses the "rigid block at the ankle" problem identified by Improving 3D Foot Motion Reconstruction in Markerless Monocular Human Motion Capture.

Relation to Prior Work

Two papers supply the method. CoTracker jointly tracks many points through occlusion. It reports 62.2 Average Jaccard on TAP-Vid-DAVIS against a prior 56.2, and over 30% better accuracy on occluded points, which matters because the stance foot is often partly hidden. The zero-velocity contact loss of Combining Recurrent Neural Networks and Adversarial Training for Human Motion Synthesis and Control penalizes foot velocity when a contact label is on. We reuse that form but apply it in image space, relative to the ground track, so it needs neither a known ground plane nor a known camera.

Three papers motivate the proposal by exposing the gap. Sim2real transfer learning for 3D human pose estimation: motion to the rescue trains on synthetic SURREAL humans. Using flow and keypoints instead of RGB, it reaches 74.7 mm PA-MPJPE on 3DPW, versus 105.6 mm from synthetic RGB, but it leaves foot sliding unconstrained. Capturing and Inferring Dense Full-Body Human-Scene Contact (BSTRO) and Shoe Style-Invariant and Ground-Aware Learning for Dense Foot Contact Estimation (FECO) show that RGB cues are needed for contact. Both need manually or capture-annotated real images, both are single-frame, and FECO must fight shoe-texture bias with style randomization. We keep RGB and drop human annotation. The supervision comes from video dynamics, so a shortcut through shoe color would not reduce the loss.

Self-supervision versus synthetic datasets is the methodological precedent. In video denoising, self-supervision on real footage beat synthetic supervision by 1.5 dB PSNR. We test whether the same "lesser evil" verdict holds for contact.

Three lines of work reach for physics instead. Contact and Human Dynamics from Monocular Video trains a 2D-keypoint contact classifier on synthetic motion capture, then optimizes centroidal dynamics. Trajectory Optimization for Physics-Based Reconstruction of 3d Human Pose from Monocular Video runs Bullet inside CMA-ES search. Towards Stable Human Pose Estimation via Cross-View Fusion and Foot Stabilization re-roots the kinematic chain at the contacting foot using multi-view pseudo-labels and a pre-fitted flat ground. All three need either 20–60 minutes per second or two of video, or labeled or calibrated contact. Our signal needs none of these.

PHUMA and CRISP are downstream consumers. PHUMA cleans web-video motion with an offline anti-skating loss. CRISP turns monocular video into simulation-ready scenes but inherits upstream contact errors from a frozen vision-language model. Neither derives contact from the pixels' own motion.

Research Program

The first stage builds the pseudo-labeler and validates it where the truth is known. We render SURREAL-style synthetic sequences with moving cameras and real Kinetics backgrounds, following the compositing pipeline of Doersch and Zisserman. On these we measure how well $\hat c_t$ agrees with true heel and toe contact, and calibrate $\sigma$. The synthetic data serves only to calibrate one scalar and to characterize failure modes, not to train the detector. That role keeps synthetic data out of the place where the domain gap hurts.

We expect four failure modes and will measure each:

  1. Motion along the viewing ray. A foot moving toward the camera shows little image motion. Ground points at the same image location share the projection but sit at a different depth, so their motion can diverge, and the residual may stay low.
  2. Stationary scenes. With a static camera and a still person, everything has zero residual. This makes a raised but motionless foot (yoga poses) look planted.
  3. Occluded ground. The ground patch next to the foot may be hidden.
  4. Moving surfaces. Escalators and treadmills move the ground itself.

The second stage trains the student. This is an RGB video contact network with a lightweight temporal head, comparable in size to FECO's ResNet-18. It is trained on large unlabeled real human video with $\hat c_t$ as soft targets, weighted by $v_t$. Frames with low visibility or near-zero total motion are dropped from the loss rather than labeled. The student therefore learns to generalize to exactly the cases the labeler abstains on. That generalization is the reason to distill instead of using the labeler directly.

The third stage uses contact for recovery. We take an off-the-shelf monocular pose estimator and add a test-time refinement that minimizes reprojection error. The refinement adds an image-space sliding term: the predicted contact probability times the residual between the projected 3D foot point and the tracked ground point. Optionally it adds a 3D zero-velocity term once a ground plane is estimated. Both terms are differentiable and cheap.

Baselines

The naive reference is the pose estimator with no contact handling. It sets the starting footskate rate and PA-MPJPE. Trajectory Optimization for Physics-Based Reconstruction of 3d Human Pose from Monocular Video reports such a kinematic baseline at 64% footskate frequency on Human3.6M.

The strongest sliding-removal method in the file is the same paper's physics optimization. It reports 8% footskate frequency on Human3.6M, and velocity jitter falling from 0.58 to 0.27 m/s, at about 20 minutes per second of video. We will run our refinement from the same kinematic initialization where reproducible, and otherwise compare against the reported number with that caveat stated. Beating it means matching or improving on 8% footskate without raising MPJPE, at under one second of compute per second of video. Matching the number at roughly a thousand times lower cost would already be a meaningful result.

For pose accuracy, Towards Stable Human Pose Estimation via Cross-View Fusion and Foot Stabilization reports 40.1 mm PA-MPJPE on 3DPW and 43.8 mm on AIST++. Our method is a refinement layer, not a new backbone, so the bar is that it should not degrade PA-MPJPE while it reduces sliding. Adding our term on top of a comparable backbone and improving sliding is the relevant claim.

For contact detection, BSTRO reports an F1 of 0.708 on RICH (HuMoR 0.314). FECO reports 0.577 on MMVP (BSTRO 0.464, DECO 0.409). Both use annotated real images. We will evaluate our student at the foot-joint level on these benchmarks with no exposure to their labels. Approaching these numbers without annotation would be the success criterion. Exceeding them after fine-tuning on the same labeled training splits, at equal backbone size and epochs, would be a stronger claim.

The sim-to-real baseline is the synthetic contact classifier of Contact and Human Dynamics from Monocular Video, which maps 2D keypoints to contact and is trained on motion capture. We pair it with an RGB variant of our student trained on rendered synthetic contact labels. Both get the same backbone, data volume and tuning budget as our student.

Two ablations isolate the key ingredient. In the first, the student is trained on pseudo-labels from thresholding the estimator's own 3D foot velocity, which isolates the value of image-relative ground tracking. In the second, the student sees only keypoints and flow rather than RGB, which isolates whether real-video supervision really allows appearance cues to help.

Why not just…?

Why not just threshold the 3D foot velocity from the pose estimator, as dataset-curation pipelines do? This is the most serious threat. PHUMA's curation shows that velocity- and height-based heuristics work well on clean motion capture. On monocular estimates, though, the foot's 3D velocity contains exactly the errors we are trying to remove: depth ambiguity accumulating down the kinematic chain, and camera motion folded into global trajectory. Labels derived from the sliding output would teach the detector to accept sliding. The ground-relative image signal is free of both error sources. The velocity-threshold ablation settles the question directly. If it matches our labels on RICH contact F1 and on Human3.6M footskate, the tracking component is unnecessary and we will say so.

Why not just use FECO or BSTRO, which already detect contact from RGB? They are strong per-frame detectors. However, they need costly real annotation (RICH capture, about 31,000 hand-labeled COFE images), are single-frame and so jitter over time, and FECO has to explicitly suppress shoe-texture shortcuts. If they turn out to be as good as our student, our contribution reduces to an annotation-free route to comparable contact. We will run each with our refinement term to separate "better contact labels" from "better use of contact."

Why not just run physics-based trajectory optimization, which already gets footskate to 8%? At 20 to 60 minutes per second or two of video, it cannot process the web-scale video that projects like PHUMA need. It also fails when initialization is poor, per the Gärtner and Rempe papers. We do not claim better physical plausibility than a simulator. We claim most of the sliding reduction at feed-forward cost, which the footskate-versus-runtime comparison measures.

Why not just train on synthetic data with heavy appearance randomization? The evidence in the file runs against this as a complete answer. Synthetic RGB transferred at 105.6 mm on 3DPW in Doersch and Zisserman. FECO needed real annotated data even with randomization. Dewil and colleagues found real self-supervision beat synthetic supervision in an analogous setting. Our synthetic-label student baseline tests this directly.

Decisive Checks

The first check comes before any training: on synthetic renders with moving cameras, the pseudo-labeler must reach high heel/toe contact precision on frames where it does not abstain. The second check runs on RICH, where ground-truth contact exists for real video, and repeats the agreement test on real footage with no synthetic shading involved. If precision is low there, the idea fails early and cheaply.

The main experiments then train the student on unlabeled real video. We evaluate contact F1 on RICH and MMVP, footskate frequency and jitter on Human3.6M, PA-MPJPE on 3DPW and AIST++, and global foot drift on EMDB. EMDB's sensors are accurate to 2.3 cm, but no reported foot-sliding result for any method on EMDB was found on Lacuna, so we will establish the baselines ourselves. MOYO serves as a stress test for the stationary-raised-foot failure mode, where we expect the labeler to abstain and the student to need help from RGB.

A final downstream check feeds refined motions into CRISP-style humanoid tracking in simulation. CRISP reports 6.9% tracking failure against 55.2% for VideoMimic. We ask whether fixing contact at the source lowers failure further on the same clips.

The proposal succeeds if the annotation-free student approaches annotated detectors on contact F1. It must also clearly beat the synthetic-label and velocity-threshold variants, and cut Human3.6M footskate toward the 8% physics result while running in near real time.