Pinned Feet: Contact as a Reparameterization of Gaussian Avatars

Systems that turn a phone video of a person into a renderable 3D avatar now work well. They usually represent the body as millions of small colored 3D blobs, called Gaussians, and the motion of those blobs is driven by a sparse "puppet": a skeleton or body mesh with a few thousand control points. Because the whole pipeline can be differentiated, the puppet and blobs are fitted by comparing rendered images to the video frames. These systems share one visible flaw. When the person walks, the rendered feet glide across the floor. Their motion comes from a body tracker that was never asked to keep a planted foot still in world space. Separate work removes foot sliding with contact labels, zero-velocity penalties, or physics simulation, but it acts only on skeletons and never looks at pixels. When the tracker misjudges a heel strike, the contact fix locks the wrong state in place.

We propose to make contact a structural part of the avatar rather than a penalty applied afterward. While a foot is in stance (on the ground and bearing weight), its control points share a single world-space position for the whole stance interval. A stance of thirty frames therefore contributes one 3D point instead of thirty, and the leg is solved to reach it. Where the stance intervals begin and end, and where each pin sits, are then refined by the same image loss that fits appearance. Rendering only small foot crops is cheap, so the video itself can vote on contact timing. The two-sentence version: pin each planted foot to one world point per stance, so the avatar cannot slide by construction, and use cheap differentiable rendering of foot regions to correct the tracker's contact mistakes instead of inheriting them.

Core Claim

Two claims make the proposal worth doing. The first is that foot sliding in reconstructed Gaussian avatars comes mainly from the upstream body trajectory, not from the appearance model. Hard pinning of sparse foot control points removes it without hurting image quality, because it also removes degrees of freedom the photometric loss could only fit to noise. The second, which carries the risk, is that pixel evidence from the avatar's own renderer can correct contact timing and pin placement when a keypoint-based contact predictor is wrong. A pipeline that runs a contact-aware tracker first and fits the avatar second cannot do this. If the second claim fails, the work reduces to a clean engineering combination, and we say so up front.

What Already Exists and Where This Differs

The rendering side supplies the machinery. GoMAvatar attaches Gaussians to the triangles of a skinned body mesh and renders at 43 FPS. It reports 30.37 dB PSNR on ZJU-MoCap. 3DGS-Avatar drives Gaussians with a learned skinning field and trains in about 30 minutes. Mango-GS and H3D-DGS show that sparse control nodes, around 0.2% of the Gaussian count in H3D-DGS, can drive dense splats stably. All four work in root-relative or unconstrained coordinates, have no notion of ground contact, and treat foot motion like any other motion. We reuse GoMAvatar's mesh binding as the appearance model, and we inherit its fixed-topology limitation. What we add is a world-space constraint on a handful of foot vertices. DEMEA cautions that local rigidity over sparse graphs over-smooths fine motion. That caution argues for confining the hard constraint to the foot sole, which is nearly rigid during stance, rather than spreading it across the body. 4DAnyone motivates the problem directly: its authors note that foot errors from the upstream body tracker get baked into the final 4D Gaussian avatar as floating and skating.

The grounding side supplies the constraint and shows its weak point. Xia et al. predict binary foot contacts and penalize foot velocity during contact on a flat floor. WHAM reconstructs world-grounded body meshes from video with a contact-aware trajectory refiner and reduces sliding on EMDB, 3DPW and RICH. Its contacts, however, come from keypoints and image features that never pass through a renderer. PhysDiff and CRISP go further and use a simulated humanoid controller to guarantee plausibility. PhysDiff reports an 86% reduction in its combined physics-error metric on HumanML3D. CRISP cuts simulation tracking failures from 55.2% to 6.9%. Both pay for a physics loop, and CRISP assumes the world is made of planes. Our proposal sits at none of these points. It does not penalize velocity; it removes per-frame foot positions during stance. It does not trust the contact predictor; it re-scores contact with rendered pixels. It does not assume a floor; each pin has its own free height, so stairs or uneven ground show up as pins at different heights.

Research Program

The model starts from a monocular video, an initial world-space body trajectory from WHAM, and WHAM's per-frame foot contact probabilities. Each foot carries three anchor vertices on the body mesh, at the heel, the ball and the toe. A stance interval $s$ for anchor $f$ spans frames $a_s$ to $b_s$ and owns one world point $q_{f,s}$. Inside the interval the anchor's position is defined, not penalized:

$$p_{f,t} = q_{f,s} \quad \text{for } a_s \le t \le b_s,$$

where $p_{f,t}$ is the anchor's world position at frame $t$. Hip, knee and ankle rotations are solved each frame with a differentiable inverse-kinematics step so the mesh reaches the pin. Outside stance, the anchors follow the skeleton freely. The pins, the free poses, the root trajectory and the Gaussian appearance are all optimized together. The objective combines GoMAvatar's photometric loss, a 2D keypoint reprojection term, and a temporal smoothness term on joint angles that discourages the knee popping that naive foot locking produces.

The interval boundaries are discrete, so they are updated by local search between continuous optimization rounds. Each candidate move shifts a boundary by one or two frames, or splits or merges an interval. A move is scored by re-rendering only the tiles that cover the two feet over the affected frames and measuring the change in photometric and silhouette error. Tile-based Gaussian rasterization makes this crop rendering a small fraction of a full frame's cost, which is what makes testing many contact hypotheses per sequence affordable. The efficiency direction enters here as a budget rather than a goal. The full method should stay within roughly 1.5 times the wall-clock training time of plain GoMAvatar on the same GPU, and we report that ratio.

The analysis asks a question no current paper answers: when the tracker and the pixels disagree about contact, which is right, and on what kinds of frames? We stratify results by the reason for disagreement, such as foot occlusion, motion blur, low camera angle and toe-only stances. This tells us whether the rendering loss actually carries contact information or merely adds noise.

Baselines

The naive reference is GoMAvatar fitted directly to WHAM's world-grounded poses with no further handling of contact. This is "do nothing" in the sense that both components are used exactly as published. It measures how much sliding survives into the rendered avatar and fixes the image-quality ceiling of the appearance model. It runs on identical frames and resolution with GoMAvatar's published training schedule.

The strongest competitor is a two-stage pipeline we build ourselves. WHAM produces contact-refined world trajectories. A Xia-style zero-velocity penalty, weighted by WHAM's contact probabilities, is added to the joint-angle fine-tuning that happens during avatar fitting. This is the obvious engineering answer, and it already contains every ingredient except pixel-driven contact revision and hard pinning. It gets the same iteration count and the same hyperparameter search budget as our method, about twenty configurations on a held-out tuning sequence, for both the penalty weight and the contact threshold. We beat it if we cut foot sliding further on ground-truth stance frames at no cost in held-out PSNR. On trajectories, the bar is lower root translation error than the WHAM-initialized pipeline. The file reports no number for WHAM's root translation error or foot sliding on EMDB, only that both drop substantially; we therefore re-run WHAM ourselves under our evaluation protocol. On rendering, the reference is GoMAvatar's reported 30.37 dB PSNR on ZJU-MoCap, which our method must match within noise.

A physics reference is PhysDiff-style projection: a simulated humanoid tracks the WHAM motion before avatar fitting. It is not a fair efficiency competitor, but it is the upper bound on plausibility. We report its sliding and its wall-clock cost to show what a simulator-free method gives up.

Three ablations isolate the key ingredients. The first keeps hard pinning but freezes WHAM's contact intervals, which tests whether pixel-driven revision matters. The second replaces hard pinning with a soft penalty while keeping the revised intervals, which tests whether reparameterization matters beyond the penalty. The third scores interval moves with 2D keypoint error instead of foot-crop rendering, which tests whether the renderer adds information that keypoints lack.

Why not just…?

Why not just run WHAM first and fit the avatar to its output? Because WHAM's contact states are only as good as its keypoints, and the file notes that refiners of this kind lock limbs in an erroneous state when a toe strike is misread. 4DAnyone's authors report exactly this kind of foot error propagating into their Gaussian avatar. This is still the alternative most likely to match us. If WHAM's contacts are right on nearly all frames of EMDB and RICH, the frozen-interval ablation will match the full method, and our contribution reduces to hard pinning. The stratified disagreement analysis settles it directly.

Why not just add a zero-velocity loss during avatar fitting? A soft penalty trades sliding against image fit and needs a weight tuned per sequence. Xia et al. also apply it on a flat floor. Pinning removes the trade-off and lets pins sit at different heights. The second ablation measures whether this matters in practice. If the soft loss matches pinning at a well-tuned weight, we will report that and treat pinning as a convenience rather than a finding.

Why not just lock the feet with inverse kinematics after reconstruction? The file notes that post-hoc locking produces jitter and knee popping when the reference motion disagrees with the ground. It also leaves the appearance fitted to the unlocked poses, so the Gaussians on the shin and foot are misregistered after locking. We measure this as the PSNR drop on foot crops after post-hoc locking of the naive baseline, and we include post-hoc locking as an extra row in the main comparison.

Why not just use a physics simulator, as PhysDiff and CRISP do? These methods give stronger guarantees, such as balance and penetration-free contact, but they put a trained controller in the loop. CRISP also assumes planar support. We position the method as the cheap, pixel-aware option and report the simulator reference so readers can see the remaining gap in plausibility.

Decisive Experiments

The main evaluation uses EMDB, whose drift-free global trajectories give world-space ground truth for foot positions, with RICH and 3DPW as further in-the-wild tests. Ground-truth stance is derived from the reference body motion with a combined foot-height and foot-velocity threshold, unless a dataset supplies contact labels. Foot sliding is the mean horizontal speed of foot anchors on ground-truth stance frames, the same quantity PhysDiff's metric measures. We report it separately for frames where WHAM's contact agrees or disagrees with ground truth. Contact accuracy is the F1 score of predicted stance intervals against ground truth. Rendering is evaluated as PSNR, SSIM and LPIPS on held-out frames, computed both on the full image and on foot crops. We use ZJU-MoCap for comparison with GoMAvatar's reported number and DNA-Rendering for harder motion. Cost is wall-clock training time on one GPU relative to GoMAvatar.

The decisive check is the disagreement subset. If pixel-revised intervals raise contact F1 and reduce sliding on frames where WHAM is wrong, and do not harm frames where it is right, the second claim holds. If not, the result is still useful as a negative finding: the renderer carries no contact information beyond keypoints, and the frozen-interval pinning method stands on the first claim alone. A secondary check uses staircase and slope segments in EMDB. There, the recovered pin heights should track the true support heights, which no flat-floor contact method can do.