Hearing Footsteps: Raw-Audio Contact Timing for Slide-Free Video Avatars

When a person is filmed walking or dancing and turned into an animatable 3D avatar, the feet are usually the worst part. The pose estimator that drives the avatar does not reliably know when a foot is planted, so the reconstructed body glides across the floor ("foot sliding") or hovers above it. The appearance model then learns blurred shoes and trouser hems, because it is asked to explain inconsistent foot positions with a single texture. Yet almost every phone video already contains a precise record of when each foot hits the ground: the footstep sound. A heel strike produces a sharp pressure transient lasting a few milliseconds. A 30 Hz video frame spans 33 ms. This proposal treats the soundtrack as a sensor for ground contact.

We will train a small network that reads the audio waveform directly, with no spectrogram step. It is supervised only by coarse, frame-level contact guesses from vision, and it learns to place contact onsets and releases at sub-frame precision. The audio-derived contact intervals then become constraints when fitting the body motion and the pose-dependent appearance of a Gaussian-splat avatar. Because the detector must run alongside rendering on a capture device, we co-design its first layers (filter length, decimation, bit-width) with a fixed on-chip memory and latency budget. Hardware co-design acts here as a constraint, not as the main contribution.

Core Claim

Audio gives contact timing that monocular vision cannot, and a raw-waveform detector can learn that timing from vision-level supervision alone. The claim is testable: on capture data with pressure-insole ground truth, the audio-plus-vision contact estimate should beat vision-only contact estimates on onset timing error. Using it as a constraint should reduce foot skating in reconstructed motion without degrading full-body rendering quality. What would be new is the supervision loop and its downstream use. A vision teacher at frame rate trains an audio student at sample rate. The student's sub-frame intervals then correct both the skeleton and the foot-region appearance of the avatar. To our knowledge from this file, no prior work uses waveform impact transients for foot contact in motion reconstruction or rendering.

Where This Sits

The motivating evidence comes from three groups of work.

Monocular avatars inherit their kinematic driver's errors. GaussianAvatar skins 3D Gaussians with SMPL poses and refines those poses jointly with appearance against RGB loss. HumanNeRF delays non-rigid deformation so that it does not absorb skeletal errors. Neither has any notion of ground contact.

Foot reconstruction from video alone is hard. FootMR shows that generic estimators produce "dead" or sliding feet. It fixes foot articulation with a dedicated lifting Transformer, but explicitly leaves true sliding to future physics-based constraints. Text-driven generators such as TEMOS acknowledge sliding as a by-product of purely kinematic modelling.

Audio-driven motion has not used footstep physics. DiscoForcing drives dance from music tokens and still relies on a robot's whole-body controller to absorb residual sliding.

Several papers supply the method we reuse. The zero-velocity contact loss of Combining Recurrent Neural Networks and Adversarial Training for Human Motion Synthesis and Control penalizes foot velocity whenever a contact label is on. We keep that loss but replace its frame-level labels, which come from annotation, with continuous-time intervals that come from sound.

SoundNet showed that a raw-waveform CNN can be trained purely by distillation from visual models on unlabeled video. Its first-layer filters became band-pass and wavelet-like. It reached 88% on DCASE against a previous best of 78%. We borrow the teacher-student structure but change the target: a temporally localized physical event instead of a scene class.

For the front-end we draw on three papers. Learning the speech front-end with raw waveform CLDNNs shows that learned waveform filters can match log-mel filterbanks. Speaker Recognition from raw waveform with SincNet restricts first-layer filters to band-pass shapes with learnable cutoffs. Raw Speech-to-Articulatory Inversion by Temporal Filtering and Decimation uses filtering-then-decimation and avoids the stationarity assumption of 25 ms windows. That last structure is also what makes a cheap streaming implementation possible.

Two works constrain the design. SoundNet itself warns that cross-modal teachers mislabel events that are off-camera. In our setting the analogue is a footstep from someone else, or music masking the step, so the student cannot simply be trusted where the teacher is silent. Physics-in-the-loop approaches such as LAGOON and CRISP prevent sliding mechanically, but at substantial compute cost. We aim for most of their contact benefit as a cheap constraint inside the existing optimization.

Research Program

Contact detector. The detector takes 16 kHz mono audio. Its front-end is a bank of sinc-parameterized band-pass filters followed by learned decimation stages. On top sit a small causal temporal convolution stack and a per-foot output head. The head emits a contact probability $p_k[n]$ for foot $k$ at a 1 kHz output rate.

Weak supervision from vision. A vision teacher produces frame-level labels $y_{k,f}\in\{0,1\}$ for each video frame $f$. The teacher is FootMR-refined poses, foot-height and foot-velocity thresholds, and the dense foot contact estimator of Shoe Style-Invariant and Ground-Aware Learning for Dense Foot Contact Estimation. We train the student with a multiple-instance objective:

$$ \mathcal{L}_{\text{MIL}} = \sum_{k,f} \text{BCE}\Big(y_{k,f},\ \tfrac{1}{\beta}\log \tfrac{1}{|S_f|}\sum_{n\in S_f} e^{\beta p_k[n]}\Big) + \lambda \sum_{k,n} |p_k[n]-p_k[n-1]|, $$

where $S_f$ is the set of audio samples falling inside frame $f$. The parameter $\beta$ controls how close the pooling is to a hard maximum. The total-variation term, weighted by $\lambda$, pushes the student to produce a few sharp transitions rather than diffuse probability.

Teacher frames where vision confidence is low are down-weighted. We add a small number of insole-labelled clips to calibrate, but they are never used for training the headline model. The core empirical question is whether sub-frame onset localization emerges from frame-level labels. It plausibly can, because the step transient is the only audio feature consistently aligned with teacher label boundaries.

Contact-constrained motion fitting. The audio contact intervals $\mathcal{C}_k$ are thresholded and fused with the visual teacher through a simple product of probabilities. They then enter pose optimization as a continuous-time zero-velocity term over a spline interpolation of the foot trajectory $\mathbf{x}_k(t)$:

$$ \mathcal{L}_{\text{contact}} = \sum_k \int_{\mathcal{C}_k} \|\dot{\mathbf{x}}_k(t)\|^2\,dt . $$

We add a ground-height term inside the same intervals. This replaces the frame-indexed version from the recurrent-adversarial work. Its effect is largest at frames that straddle a strike, where frame-level labels are ambiguous.

Contact-aware appearance. In GaussianAvatar's joint motion-appearance optimization, we (a) apply $\mathcal{L}_{\text{contact}}$ to the SMPL foot joints and (b) add a contact-phase input to the dynamic appearance network, but only for Gaussians attached to the shank and foot. The contact-phase input is time since the last strike, decayed. This follows the observation of Choi et al. and Renke Wang that cloth state depends on motion history rather than instantaneous pose. Contact phase is a physically meaningful piece of that history, which pose velocity encodes only indirectly.

Co-designed front-end. The detector should cost almost nothing next to a renderer. We therefore search jointly over filter count, filter length, decimation factors and 8- versus 16-bit fixed point, against two constraints. The first is an on-chip buffer budget: the receptive field of audio history held in SRAM. The second is a per-frame latency budget. The search follows the joint model-hardware search spirit of Learned Hardware/Software Co-Design of Neural Accelerators. It is deliberately small: the outcome is a Pareto curve of onset error versus memory and latency on a mobile CPU or DSP, not a new accelerator.

Data

Existing benchmarks give partial coverage.

  • People-Snapshot and NeuMan (used by GaussianAvatar), ZJU-MoCap (HumanNeRF), and MOYO and RICH (FootMR) provide pose and rendering evaluation. We will check which releases include usable audio. MOYO yoga contains few strikes and is mainly a test that the method does no harm.
  • AIST++ and FineDance are dance with music. They test robustness to music masking, using the music stem when separable.

Because no file resource offers footstep audio with ground-truth contact, we will record a calibration and evaluation set. It consists of synchronized monocular phone video, a reference multi-view rig, a phone microphone, and pressure insoles. It covers several floor materials (wood, tile, carpet), footwear types, gaits (walking, running, turning, intentional slides), and background conditions (quiet, speech, music). Carpet and socks are included on purpose as a failure regime.

Baselines

Do nothing. The naive reference is GaussianAvatar trained exactly as released, driven by its own jointly refined SMPL poses with no contact term. This shows how much skating and foot-region blur the standard pipeline leaves. The file reports 35 FPS rendering on People-Snapshot for GaussianAvatar and better PSNR, SSIM and LPIPS than prior methods, but gives no numeric image-quality values. We will therefore re-run it ourselves on identical splits.

Strongest vision-only competitor. This is FootMR-refined poses plus the frame-level zero-velocity loss, with contact labels from the vision teacher. It is the strongest baseline because it combines the best foot articulation available in the file with the standard contact loss. It differs from our system only in where contact comes from. FootMR reports a 30% reduction in ankle joint angle error on MOYO over the best video-based method and over 50% better 2D foot alignment on MOOF, at about 10 ms of added runtime. These are the numbers our pipeline must not regress on when we keep FootMR in the loop.

Contact detection baselines. Here we compare against the vision teacher alone, a classical energy-onset detector on the waveform, and log-mel CNNs at 10 ms and 2 ms hops with matched parameter count. All are trained under the same weak labels and tuning budget. No reported result was found on Lacuna for audio foot-contact onset timing, so all detection numbers are our own.

Physics reference. A physics-tracking reference in the style of LAGOON or CRISP, applied to the same vision poses, gives an upper reference for skating removal and a cost comparison. No reported foot-skate number for these was found on Lacuna.

Ablations. Two ablations isolate the key ingredient. In the first, we feed the motion fitter the student's predictions snapped to frame boundaries. If the gain vanishes, sub-frame timing is what matters. In the second, we remove the contact-phase input from the appearance network while keeping the contact-corrected poses. This separates the skeletal fix from the appearance conditioning.

Fairness and success criteria. All methods use the same frames, the same SMPL initialization, the same number of optimization steps, and a matched hyperparameter sweep. Beating the vision-only baseline means two things on the insole-labelled set: lower foot-skate velocity during ground-truth contact, and lower onset error. The second must hold with confidence intervals that exclude zero across subjects. Full-frame PSNR and LPIPS on held-out frames must not drop.

Why not just…?

Why not just use a spectrogram onset detector with a short hop? This is the most serious threat. A log-mel or spectral-flux detector with a 2 ms hop may localize a heel strike nearly as well as a learned waveform front-end. We expect raw waveforms to matter less for when a strike happens and more for what kind of contact it is: heel versus toe, a slide's friction burst versus a strike, and discrimination under music. Those cues live in fast transients that fixed windows smear, as argued in the articulatory-inversion and SVDD 2024 work. The matched-parameter comparison above settles this. If the 2 ms mel model matches the raw model on onset error, slide/strike F1 and music-condition robustness, we will report that the gain comes from audio contact itself. The front-end then becomes a cost choice decided by the co-design curve.

Why not just snap feet with a velocity threshold and inverse kinematics? Threshold snapping produces popping and knee locking during turns and intentional slides, as noted in the contact and physics literature cited above. More fundamentally, it inherits the 33 ms ambiguity of the poses it thresholds. We include threshold-plus-IK on FootMR poses as an extra baseline. The intentional-slide clips in our dataset are designed to expose it, since a threshold cannot tell a deliberate glide from skating.

Why not just film at higher frame rate or trust vision contact estimators? High-speed capture removes the timing argument, and we will measure that on a 120 Hz subset to quantify how much of our gain it recovers. However, the target regime is ordinary phone video and existing footage, where the soundtrack comes for free. Dense visual contact estimation still fails when feet are occluded, motion-blurred or out of frame. The occluded-foot slice of the evaluation tests whether audio helps exactly there.

Why not just run a physics tracker? Physics tracking removes sliding by construction but adds simulation and controller cost, and can distort stylized motion. We report skate reduction per millisecond of added compute against the physics reference. If physics wins on both axes, our contribution narrows to cheap contact labels that a physics tracker could also consume. We would test that combination directly.

Proposed Experiments and Decisive Checks

Contact detection. On the insole set we report median and 90th-percentile onset and release error in milliseconds, per-foot contact F1, and slide-versus-strike F1. Results are broken down by floor, footwear and noise condition. The first decisive check is whether the weakly supervised student achieves onset error clearly below half a video frame (about 16 ms) while the vision teacher does not. If it does not, the core claim fails.

Motion. On the insole set and on whichever of RICH, MOYO and AIST++ carry audio, we report:

  • mean foot horizontal speed during ground-truth contact;
  • ground penetration and floating height;
  • FootMR's ankle joint angle error;
  • full-body joint error, to confirm the contact term does not trade the rest of the body for the feet.

Rendering. On People-Snapshot, NeuMan, and our captures, we report PSNR, SSIM and LPIPS on full frames and on foot-and-shank crops. We add a temporal flicker measure on the foot region and a small paired user study on foot realism. The second decisive check is the frame-snapped ablation. If sub-frame timing does not beat frame-snapped labels on skating and foot-crop LPIPS, the audio contribution reduces to a contact detector that vision could eventually match.

Cost. We plot onset error against SRAM footprint and per-frame latency for the co-designed front-end on a mobile processor. The requirement is that the chosen point adds well under FootMR's reported 10 ms to the pipeline. If the cheapest useful configuration costs more than that, the deployment story fails even if accuracy holds.