Systems that recover 3D human motion from a single video often produce feet that glide across the floor while they should be planted. This artifact is called foot sliding or foot skating. The usual fixes have two problems. One family is offline and heavy: a physics simulator tracks the estimate. The other family is blunt: the foot is snapped in place whenever its speed drops below a threshold. Both treat every horizontal foot motion during contact as an error. That makes them erase real slides, such as dance pivots or a slip on a wet floor, along with the artifacts.
We propose a recursive filter that runs online at constant cost per frame. It keeps the floor plane, the camera, and each foot's slip velocity inside its estimate of the world. The key observation is simple. Once the filter has an estimate of the floor plane, the 2D image location of a planted foot pins down the foot's full 3D position, because the camera ray through that pixel meets the floor at one point. Foot slip then stops being something a threshold has to guess. The image either supports the motion or it does not.
Core Claim
An extended Kalman filter (EKF) predicts the state forward with a motion model, then corrects it whenever a measurement arrives, weighting each source by its uncertainty. We claim that an EKF can separate artifact sliding from true sliding in streaming monocular motion capture, provided it does three things.
First, it augments its state with the floor plane and with a per-foot tangential slip velocity. Second, it fuses two kinds of measurement separately. One is the 3D pose from a learned lifter, which is complete but prone to drifting in depth and root position. The other is the 2D foot keypoints, which are precise in the image plane but carry no depth. Third, it lets a learned contact probability control how strongly slip velocity is pulled toward zero, instead of imposing a hard zero-velocity constraint.
The prediction is that this removes most skating without the high-frequency jitter of simulation-based correction. It should also preserve slides the image actually shows, which threshold locking and zero-velocity penalties cannot do by construction.
Relation to Prior Work
Two lines of work motivate the proposal by showing the gap.
The first is physics-based correction. MultiPhys feeds kinematic video estimates into MuJoCo with a PPO tracking policy. It cuts skating substantially but raises acceleration error and runs offline. Skeleton2Stage fine-tunes dance diffusion models against physics rewards and has to add an anti-freezing reward, because optimizing for foot adherence makes motion go still. Neither method uses image evidence to decide whether a given slide is real. Both are batch procedures.
The second line is offline geometry. PHUMA's PhySINK cleans motion with anti-skating inverse kinematics and estimates the ground by batch majority vote. We move the ground estimate into a recursive state that is refined online.
Three filtering papers supply the method.
- Tight coupling of a learned measurement. TLIO fuses a network's displacement predictions, together with their predicted uncertainties, as EKF measurements. We fuse the lifter's 3D pose the same way, letting it carry its own covariance.
- Mode-dependent constraints. Skog et al. handle motion modes in foot-mounted navigation with a bank of Kalman filters whose cost grows exponentially. We replace the discrete bank with one filter whose process noise is a continuous function of contact probability.
- Sliding as a filtered state. Piga et al. show that an EKF can track continuous sliding velocity of in-hand objects from tactile derivatives. We borrow the idea of sliding velocity as a state and apply it to full-body foot contact, where sticking, slipping, and swing alternate.
Two filtering papers constrain the design.
- He et al. obtain observability of a robot's yaw on a moving platform only by instrumenting the floor. This is a reminder that contact constraints without an external reference leave some directions unobservable. Our external reference is the camera.
- Osman et al. embed multibody dynamics in an iterated EKF, but only for fixed-base systems. The cost of their Jacobians argues against full dynamics here. We keep the body model kinematic and reserve iteration for the contact feet.
The Filter
The state contains the following:
- the root pose and velocity;
- the joint angles;
- for each foot, a 3D position and a tangential slip velocity $s_t$ in the floor plane;
- the floor plane $(\mathbf{n}, d)$, with unit normal $\mathbf{n}$ and offset $d$, under a static camera with known intrinsics.
The slip velocity follows a mean-reverting process whose pull toward zero depends on the contact probability $p_t \in [0,1]$ from a small learned detector:
$$ s_{t+1} = \big(1 - \beta\, p_t\big)\, s_t + w_t, \qquad w_t \sim \mathcal{N}\!\big(0,\ \sigma^2(p_t)\, I\big). $$
Here $\beta \in (0,1]$ sets the reversion rate. The function $\sigma(p_t)$ is small when contact is likely and large during swing. Under confident contact, slip is strongly expected to be zero. It can still grow if measurements keep demanding it, which is what separates this from a zero-velocity constraint.
There are three measurements.
- The lifter's 3D joint positions, with a covariance that is learned and inflated along the camera depth axis, where lifters drift.
- The 2D foot keypoint, projected through the camera model.
- A soft floor-contact pseudo-measurement $\mathbf{n}^\top \mathbf{f}_t + d = 0$ on foot position $\mathbf{f}_t$, with variance $r_0 / (p_t + \epsilon)$. The constant $r_0$ is a base variance and $\epsilon$ prevents division by zero.
During contact, measurement 3 ties the foot to the plane, and measurement 2 then fixes its position on that plane. A depth or root drift in the lifter disagrees with the 2D evidence, and the gain assigns most of the correction to the lifter. A real pivot moves the 2D keypoint consistently, so slip velocity grows to explain it. Floor penetration and floor tilt errors show up as innovations that update $(\mathbf{n}, d)$ through the cross-covariances.
Research Program
We will build the filter on top of a frozen per-frame video lifter. The lifter is the stand-in for any upstream estimator.
The learned parts are small:
- the contact-probability head;
- the lifter's covariance head, trained by Gaussian negative log-likelihood as in TLIO;
- the noise schedules $\beta$ and $\sigma(\cdot)$, tuned by backpropagating through the unrolled filter in the manner of Piga et al..
Contact labels and ground-truth slip come from mocap in AMASS and from the ground truth in RICH and Hi4D.
We will also analyze observability. We will characterize which directions of the floor parameters and the root state become observable after one or two contact events. We expect yaw about the floor normal to remain unobservable from contact alone, mirroring He et al., but to become observable from the camera. A secondary, explicitly weaker test applies the filter to generated motion, where there is no image. In that case the filter reduces to a prior-driven smoother, and we expect it to remove sliding without telling real slides from artifacts.
Baselines
Naive reference. The raw lifter output, with no correction.
Heuristic reference. Threshold foot locking with leg inverse kinematics. It uses the same contact detector cut at a tuned threshold, so any difference comes from soft versus hard contact handling rather than from detection quality. Beating it means lower skating at equal or better MPJPE, plus lower velocity error on frames where the ground-truth foot is truly sliding.
Strongest existing correction for video estimates. MultiPhys, which reports foot skating reduced by 35% on Hi4D and by 137% on ExPI relative to SLAHMR, along with increased acceleration error. No acceleration-error figure is reported on Lacuna. We will run our filter causally on the same SLAHMR outputs over Hi4D and ExPI, using MultiPhys's skating metric. Beating it means matching its skating reduction while lowering, rather than raising, acceleration error, at a fraction of its runtime. If we only match skating at much lower cost, we will report that as the result.
Strongest offline geometric method. PhySINK, which reports improving non-skating rates from ~50% to ~90%. We will run it on the same clips with access to the full sequence, as an upper reference that sees the future.
Generated motion. The reference is Skeleton2Stage, which reports Foot-Ground Deviation on AIST++ for EDGE dropping from 11.85 to 2.51. We will apply our filter to untuned EDGE samples and measure the same metric. We do not expect to beat a generator fine-tuned for this goal. The question is how close a training-free post-filter gets.
Ablations isolating the key ingredients.
- Replace slip-velocity states with a soft zero-velocity pseudo-measurement, using the same $p_t$-scaled variance. This tests whether letting slip persist is what preserves real slides.
- Freeze the floor at its first-frame estimate. This tests the plane state.
- Drop the separate 2D keypoint measurement. This tests image-based separation of real from artifact slides.
All variants share the lifter, the contact detector, the data splits, and an equal hyperparameter search budget.
Why not just…?
Why not just lock the foot and solve inverse kinematics? Locking cannot represent a planted foot that is really moving, so it fails on pivots by construction. A false-positive contact freezes the foot in mid-air, the failure noted for threshold detectors in this pedestrian navigation work. The true-slide velocity metric below measures this directly. If locking ties our filter on ordinary walking, we will say so; the argument rests on dynamic sequences.
Why not just run a physics simulator? MultiPhys shows that simulation reduces skating but adds jitter and runs offline. Skeleton2Stage reports simulator projection at 30 s per sequence. A simulator also has no access to image evidence about whether a slide happened. The comparison on Hi4D and ExPI settles the accuracy question.
Why not just run a fixed-lag sliding-window optimization with the same costs? This is the real threat. A fixed-lag optimizer with the same plane, slip, and keypoint terms is a more accurate solver of the same model, and it may match or beat the EKF at modest latency. We will run it with a window matched to our latency budget. If it wins, the contribution is the state model, not the EKF as an inference engine, and we will say so plainly. The EKF's remaining case is constant per-frame cost and calibrated covariance, and we will measure both.
Why not just fine-tune the motion model? Fine-tuning, as in Skeleton2Stage, cannot use test-time image evidence and has to be redone for each domain. For video capture this is a different tool. For generated motion, it is the stronger option, and our secondary experiment only asks how close a post-filter gets.
Proposed Experiments
The primary evaluation runs on RICH, Hi4D, and ExPI, restricted to static-camera sequences. It reports the following:
- MultiPhys's foot-skating metric;
- ground penetration;
- acceleration error;
- MPJPE, keeping in mind HumanTracker's finding that MPJPE is blind to skating;
- per-frame runtime;
- 3σ consistency of the filter's covariance, as in TLIO.
First decisive check: telling real slides from artifacts. On ground-truth contact frames with non-zero true foot velocity, mined from mocap and dominated by pivots, we compare predicted versus true foot velocity. The full filter should track these slides. The zero-velocity ablation and the threshold-locking baseline should suppress them. If the full filter suppresses them too, the central claim fails.
Second decisive check: robustness to a tilted floor. We perturb the initial floor estimate by several degrees on RICH sequences. The plane state should recover within a few contact events, while the frozen-plane ablation should show persistent penetration or hovering.
Secondary check: generated motion. On AIST++, we apply the filter to EDGE samples and report Foot-Ground Deviation against Skeleton2Stage's reported 2.51, without retraining.
References
- MultiPhys: Multi-Person Physics-aware 3D Motion Estimation
- Skeleton2Stage: Reward-Guided Fine-Tuning for Physically Plausible Dance Generation
- PHUMA: Physically-Grounded Humanoid Locomotion Dataset
- TLIO: Tight Learned Inertial Odometry
- Tightly Integrated Motion Classification and State Estimation in Foot-Mounted Navigation Systems
- A Differentiable Extended Kalman Filter for Object Tracking Under Sliding Regime
- Legged Robot State Estimation within Non-inertial Environments
- A Tightly Coupled IMU-Based Motion Capture Approach for Estimating Multibody Kinematics and Kinetics
- AMASS: Archive of Motion Capture As Surface Shapes
- A Novel Pedestrian Navigation Algorithm for a Foot-Mounted Inertial-Sensor-Based System
- HumanTracker: Towards Comprehensive and Human-Aligned Motion Tracking Benchmark