Exocentric Cameras as a Training-Time Verifier for Foot Contact in Egocentric Pose

Head-mounted cameras such as smart glasses almost never see the wearer's legs. Systems that estimate full-body motion from these cameras must therefore guess what the feet are doing, and their guesses slide. A planted foot glides along the floor because the model learned to imitate motion statistics, not the fact that a foot carrying body weight does not move. Datasets such as Ego-Exo4D record each egocentric session alongside four or five fixed third-person ("exocentric") cameras that see the feet clearly. Current egocentric pose models use those cameras at most indirectly, through the 3D annotations derived from them. We propose to use them as a checker. During training, the exocentric views decide automatically whether each foot in a sampled egocentric motion is planted where and when the cameras say it is. The egocentric model is then fine-tuned with reinforcement learning against that pass/fail signal. At test time it runs on the glasses alone.

This turns foot sliding into a problem of the type where reinforcement learning has recently worked best. In reinforcement learning with verifiable rewards, a cheap programmatic checker, such as a math answer matcher or a code test, grades each sampled output, and that is what makes training stable (One Training Example). Continuous perception tasks usually lack such a checker and fall back on learned judges that can be gamed. Calibrated, static third-person cameras give us a checker that is geometric, computed from raw evidence, and requires no human labels. A curriculum over how trustworthy that checker is on each clip keeps the reward informative early in training.

Core Claim

Foot sliding in egocentric pose estimators is a property of the motions the model samples, not of its training targets. Kinematic models trained on clean ground truth still slide: EgoEgo and DSPoser are both diffusion generators (models that produce samples by gradually denoising random noise) whose losses treat a planted foot like any other joint.

We claim that rewarding the model on its own samples fixes sliding where supervising it on pseudo-labels does not. Concretely, a pass/fail reward computed from multi-view exocentric evidence, optimized online with group-relative policy optimization, should reduce foot sliding on the Ego-Exo4D Ego-Pose test set substantially more than supervised training on exocentric-triangulated pseudo-labels from the same unlabeled clips. It should do so without increasing joint position error. (Group-relative policy optimization, or GRPO, samples several outputs per input and pushes the model toward those that score above the group average.) The reward needs no 3D annotation, so it can be applied to any synchronized ego-exo footage, including footage outside the annotated subset.

The closest existing work differs from this in identifiable ways. CAMVERSE showed that a geometry engine scoring generated video can serve as a verifiable reward for GRPO on a diffusion model, but it controls camera trajectories, not articulated bodies, and its verifier inspects the generated output itself rather than a second sensor stream. PhysDiff inserts a physics simulator into the sampling loop to project motions onto plausible ones. It changes how the model is sampled, not its weights, and it pays simulator cost on every generation. 3D Ego-Pose Estimation via Imitation Learning enforces physical contact by having the egocentric model drive a simulated humanoid with joint torques, but it is limited to cyclic locomotion and never sees the real contacts. WHAM predicts foot contact from third-person video, where feet are visible, and does not transfer that ability to a camera that cannot see them. Our proposal borrows CAMVERSE's training recipe and supplies the missing ingredient for bodies: a verifier that observes the contacts the policy cannot observe.

The Verifier

Let $\hat{p}^f_t$ be the 3D position of foot $f$ at frame $t$ in a sampled motion, expressed in the world frame shared by the Aria glasses' tracking and the calibrated exocentric cameras. From each exocentric camera $k$ we run an off-the-shelf 2D keypoint detector to get foot detections $u^f_{t,k}$ with confidences.

Because the GoPros are stationary, the verifier can detect planting directly in pixels. A foot is labeled planted, $c^f_t = 1$, when its detection stays fixed within a small pixel radius over a short window in at least two confident views. Using two views guards against occlusion and detector noise.

A sampled window of motion passes for foot $f$ when two conditions hold:

$$ \mathrm{pass}^f = \mathbb{1}\Big[\max_{t,k\in V_t}\|\Pi_k(\hat{p}^f_t) - u^f_{t,k}\| < \tau\Big]\cdot \mathbb{1}\Big[\max_{t:\,c^f_t=1}\|\Delta \hat{p}^{f,xy}_t\| < \epsilon\Big]. $$

Here $\Pi_k$ is camera $k$'s projection into its image, $V_t$ is the set of views where the foot is confidently detected, $\tau$ is a reprojection tolerance in pixels, and $\Delta \hat{p}^{f,xy}_t$ is the horizontal frame-to-frame displacement of the foot, which must stay below $\epsilon$ wherever the cameras say the foot is planted. The reward for a window is the fraction of foot-windows that pass, so every frame contributes binary checks. Coarse joint terms from the same reprojection test cover the rest of the body.

The two conditions protect each other. The stationarity term alone could be gamed by freezing the feet, but frozen feet fail reprojection during swing phases. The reprojection term alone tolerates small drift during contact, which is exactly the artifact we target. We set $\tau$ and $\epsilon$ on the annotated training split, choosing values at which the verifier's verdicts agree with a sliding measure computed from ground truth. Measuring this agreement is part of the study, not an assumption.

Research Program

The policy is a DSPoser-style conditional diffusion model. It takes the Aria head trajectory and the egocentric video as input, is pretrained on AMASS and the annotated Ego-Exo4D split as in DSPoser, and is then fine-tuned with GRPO over its denoising chain, following CAMVERSE. Exocentric streams are read only by the verifier. The policy never receives them, so the deployed model is egocentric-only.

The curriculum targets the specific way GRPO fails here. When every sample in a group scores the same, whether all pass or all fail, the update carries no information. Early in training, clips with non-cyclic contacts and poor exocentric visibility produce exactly that. We therefore order clips along two axes: how reliable the verifier is (how many views confidently see both feet, and whether the floor is flat according to the Aria point cloud) and how complex the contacts are (standing, then cyclic gait, then turning and pivoting, then the sports and dance scenarios). We advance along the curriculum when the variance of group rewards on the current tier falls, in the spirit of policy-driven pacing in Actor-Curator and DRIVE, and we replay earlier tiers to limit forgetting.

The curriculum is a supporting part, not the main claim. It is motivated by the collapse that 3D Ego-Pose Estimation via Imitation Learning reports on non-cyclic motion and that CRISP reports on unregularized monocular inputs. If it fails to help, the main claim survives.

The analysis has three parts. First, we validate the verifier on annotated clips. Second, we track over training how often the policy's samples fail verification by contact type. Third, we run a reward-hacking audit using metrics the reward never sees.

Baselines

The naive reference point is DSPoser as published, with no further training. On Ego-Exo4D it reports an MPJPE of 16.84 cm (mean per-joint position error, the average distance between predicted and true joints; DSPoser). We found no reported foot-sliding number on Lacuna for DSPoser on Ego-Exo4D, so we will measure its sliding ourselves with a foot-skating metric computed from ground-truth contacts. Our reproduction must land close to 16.84 cm before any comparison counts.

The strongest competitor is pseudo-label distillation. We triangulate exocentric 2D keypoints into 3D poses on the same unlabeled clips the RL arm uses, derive contact labels from the same stationarity rule, and fine-tune DSPoser with its standard loss plus a zero-velocity penalty on planted feet, the contact-loss technique of Xia et al.. This baseline uses identical exocentric information and differs only in using it as targets rather than as a sample-level reward, which makes it the most direct test of our claim. It receives the same clips, the same number of gradient steps measured in GPU-hours, and the same hyperparameter search budget. Our method beats it if it lowers ground-truth foot skating on the Ego-Exo4D test split by a margin larger than seed variance, measured over three seeds, while keeping MPJPE within that variance.

The inference-time competitor is DSPoser followed by contact prediction and inverse-kinematics foot locking. Inverse kinematics solves for joint angles that pin each planted foot in place. We also run a WHAM-style contact-aware trajectory refinement step (WHAM) trained on the same data. We compare on sliding, on jitter (joint acceleration error), and on runtime.

The judge competitor replaces our verifier with a learned scorer that sees only the predicted motion and is trained on annotated clips to predict ground-truth sliding. It is then optimized with the same GRPO setup. This isolates whether the checkable, evidence-based reward matters, as opposed to having any reward at all.

The physics competitor uses the success of a simulated humanoid imitating the motion as the reward, in the manner of CRISP and 3D Ego-Pose Estimation via Imitation Learning. This tests a verifiable reward that checks plausibility but not agreement with what actually happened.

Two ablations isolate the key ingredients. The first runs GRPO with the full verifier but samples clips uniformly instead of by curriculum. The second drops either the stationarity term or the reprojection term from the verifier.

Why not just…?

Why not just triangulate the exocentric views into pseudo-labels and train with supervision? This is the real threat, and it is our main baseline rather than something we argue away. Our reason to expect it to fall short is that supervised denoising losses act on targets, while sliding appears in samples. EgoEgo and DSPoser slide despite training on clean motion-capture data. If pseudo-label distillation with a contact loss matches our sliding numbers, the RL framing adds nothing and we will report that. The comparison is matched in data and compute and settles the question.

Why not just lock the feet with inverse kinematics after prediction? PhysDiff reports that post-hoc fixes to physically unrealistic motion produce "jerky, unnatural movements", and WHAM needed a learned refinement network precisely because flat-ground locking fails on stairs. Post-processing also cannot correct the upper body, which keeps drifting while the feet are pinned. We measure jitter and knee-angle anomalies alongside sliding. If locking matches our sliding at equal jitter, the honest conclusion is that egocentric sliding is a post-processing problem.

Why not just train a reward model or a vision-language judge to spot sliding? A judge that sees only the predicted motion learns which motions look plausible, not where this person's feet actually were. That makes it open to the reward exploitation that ReAlign had to counter with retrieved real motions as anchors. CAMVERSE's gains came from replacing judgment with geometric measurement. The judge arm tests whether the same holds for bodies. The key signature to look for is rising judge reward alongside a flat or worsening ground-truth sliding metric.

Why not just reward physical plausibility in a simulator? A simulator can certify that a motion could happen, but not that it did. A plausible, balanced stance at the wrong location passes. Simulator rollouts are also far more expensive per sample than reprojection checks, which matters when GRPO scores many samples per clip. The physics arm shows whether agreement with real evidence buys accuracy beyond plausibility. A combined reward is a natural follow-up if both terms help.

Proposed Experiments and Decisive Checks

All training uses Ego-Exo4D, with egocentric-only evaluation on its Ego-Pose test split. We report MPJPE, MPJVE (the velocity analogue of MPJPE), a foot-skating rate computed from ground-truth contacts, joint acceleration error, and ground penetration.

To test transfer to a different domain, we evaluate on the synthetic ARES benchmark without fine-tuning on it. There EgoEgo reports 121.1 mm MPJPE against 141.6 mm for the next best method (EgoEgo). We will also measure foot skating there. We found no reported results on Lacuna for Kinpoly-MoCap or GIMO, so any use of them would be exploratory.

Four checks decide the outcome. First, the verifier must be valid. On annotated clips, its pass/fail verdicts should agree with ground-truth sliding well above chance in every activity domain. If they do not in some domain, we restrict claims to the domains where they do. Second, we run the matched comparison against pseudo-label distillation on sliding and MPJPE across seeds, which tests the core claim. Third, the reward-hacking audit: improvements must appear on the ground-truth skating metric and on ARES, neither of which the reward touches, and must not come with rising jitter. Fourth, the scaling check: we hold the annotated data fixed and vary the amount of unlabeled ego-exo footage given to the verifier. If sliding keeps falling as footage grows, this shows that exocentric cameras can replace 3D annotation as the source of contact supervision for glasses-only pose estimation. The curriculum ablation is reported alongside, read as evidence about training stability rather than as the headline.