Putting the Wearer Back Into Egocentric 4D Reconstruction

Methods that rebuild a 3D scene from head-mounted video currently treat the wearer's own arms and hands as noise. A 2D segmenter cuts them out, the loss ignores those pixels, and the method reconstructs only the room and the objects that move in it. Methods that estimate the wearer's body take the opposite shortcut. They infer the unseen body from how the head moves and never reconstruct the room. We propose to do both at once. The wearer's body becomes an explicit, renderable part of a time-varying 3D model, alongside a static background and rigidly moving objects. The fit is done jointly, so that each part constrains the others. Visible hand pixels correct the body pose. The body's contact with objects steers object tracking while the hand hides the object. The objects' current positions, rather than a frozen map of the room, tell the body where it can and cannot be.

The opportunity matters because the two communities have been throwing away each other's most useful signal. EgoAllo shows that anchoring hand estimates to a full-body skeleton cuts world-frame hand error from 237.9 mm to 131.5 mm. Yet the scene reconstructors that most need stable hands, such as EgoGaussian, discard the body entirely. The synthetic-to-real gap enters in two ways. First, as a design constraint: the pipeline should learn nothing from rendered images and should fit real pixels at test time. Second, as a measurement tool: the Aria Digital Twin provides real recordings paired with pixel-aligned synthetic renders of the same trajectories. That pairing lets us separate the appearance gap from the geometry and motion gap, and it lets us score what lies behind the wearer's hand, which no real capture can reveal.

Core Claim

We claim the following. Reconstruct an egocentric video as three coupled layers of 3D Gaussians: a static background, per-object rigid Gaussians, and body Gaussians attached to the surface of an articulated body model driven by a learned motion prior. (3D Gaussians are small colored, semi-transparent blobs whose projection can be rendered and optimized by gradient descent.) Fit these layers with a single photometric objective plus contact terms against the scene as it is at each moment. This joint fit will beat a cascade of the best separate scene and body systems in three places: object tracking during hand occlusion, hand and body pose near contact, and background completion behind the wearer. Because the geometry is fit to real pixels rather than regressed by a network trained on renders, the gap between synthetic-input and real-input error should also be small.

Two ingredients are new relative to the file. The self-occlusion mask is produced by rendering the body layer itself, not by an external 2D segmenter. Scene grounding also uses the time-varying object layer rather than a static structure-from-motion map. Scene-aware egocentric pose work by Wang et al. shows why scene grounding is worth having: on EgoPW-Scene, MPJPE fell from 189.6 mm to 118.5 mm and contact accuracy rose from 38.8% to 89.4%. However, that work requires a static scene reconstructed beforehand, so it cannot handle a drawer that opens or a box that is carried away.

Research Program

The representation extends EgoGaussian's split of background clips and dynamic clips, which motivates and supplies our object layer. EgoGaussian keeps manipulated objects as rigid Gaussian sets with a per-frame six-degree-of-freedom pose. That choice reached 30.33 dB dynamic-frame PSNR on HOI4D against roughly 25.34 dB for an unconstrained deforming 4DGS baseline. Our change is to add a third layer. Body Gaussians are bound to an SMPL-H mesh, a standard parametric model of the human body and hands, posed by joint angles $\theta_t$ at frame $t$. The pose is initialized from samples of EgoAllo's head-conditioned diffusion prior, which is trained on the AMASS motion-capture corpus and is therefore real motion data, not rendered data. The fit then minimizes

$$ \mathcal{L}=\sum_t \big\| I_t - R_t(G_{bg},\{T_k(t)\,G_k\},G_{body}(\theta_t)) \big\|_1+\lambda_c\,\mathcal{L}_{contact}(\theta_t,\{T_k(t)\})+\lambda_p\,\mathcal{L}_{prior}(\theta_{1:T}\mid h_{1:T}). $$

The symbols are defined as follows:

  • $I_t$ is the observed frame.
  • $R_t$ is the Gaussian renderer at the camera pose for frame $t$.
  • $G_{bg}$ and $G_k$ are the background Gaussians and those of object $k$.
  • $T_k(t)$ is object $k$'s rigid pose at frame $t$.
  • $G_{body}(\theta_t)$ is the posed body layer.
  • $h_{1:T}$ is the head trajectory from visual SLAM.
  • $\mathcal{L}_{prior}$ penalizes deviation from the diffusion prior's plausible motions. We implement it as a denoising-score regularizer, not EgoAllo's Levenberg-Marquardt guidance, and we accept offline speed.

The contact term does two jobs. It penalizes body Gaussians that penetrate the background or any object at its current pose. During detected grasps, which we define as hand–object distance falling below a threshold for several frames, it also ties the object's rigid velocity to the velocity of the hand contact points. This coupling is what we expect to rescue object tracking when the hand covers most of the object, which is the regime where HOI4D reports BundleTrack collapsing from about 86% on synthetic data to 19.3% on real captures.

Because the body layer is rendered, its alpha channel (per-pixel opacity) marks exactly which pixels the wearer occludes. Background Gaussians therefore never receive gradients from limb pixels, and no EgoHOS mask is needed after the first iterations. To avoid the floating-Gaussian failure that RobustSplat attributes to densification driven by transients, we adopt its supplied technique of delaying densification until the static layer stabilizes.

We state two limitations up front. Forward-facing cameras see the limbs rarely: Jiang et al. report under 10% of daily activity. Photometric evidence about the body therefore comes almost entirely from hands and forearms during manipulation. Elsewhere the body is inferred from the prior plus contact. We will report results separately for manipulation segments and walking segments. We also do not model hand shadows, which RobustSplat and Split4D both leave open. Split4D's blurred contact boundaries are exactly the failure our explicit layer assignment is meant to prevent.

Baselines

The naive reference point is static 3D Gaussian Splatting with EgoHOS hand masks and RobustSplat-style delayed densification. It asks how far masking alone goes. It should reconstruct the background reasonably well but cannot track objects or the body. We run it with the same camera poses, iteration count and resolution as our method.

The strongest existing scene method is EgoGaussian, with 30.33 dB dynamic-frame PSNR on HOI4D. It is the closest competitor because it already has the rigid object layer and differs only in discarding the wearer. We rerun it on our HOI4D and ADT splits with its released preprocessing and the same optimization budget.

The strongest body methods are EgoAllo and EgoEgo. EgoEgo reports 121.1 mm MPJPE on the synthetic ARES benchmark against 141.6 mm for the next baseline. EgoAllo reports 131.5 mm world-frame hand MPJPE with its body prior. On ADT, which EgoAllo also evaluates, no reported body MPJPE result was found on Lacuna. We therefore rerun EgoAllo ourselves with identical SLAM input. The scene-grounded competitor is Wang et al.'s scene-aware estimator (118.5 mm MPJPE, 89.4% contact, 84.1% non-penetration on EgoPW-Scene). We will adapt its static-scene grounding to Aria's forward cameras by conditioning the EgoAllo prior on a single static reconstruction built beforehand. This is not the original downward fisheye system, and we will say so.

The key baseline is the cascade. It runs EgoGaussian and EgoAllo independently and then applies a post-hoc penetration correction against EgoGaussian's reconstructed scene. It uses the same inputs and total GPU hours as our joint fit.

Three ablations isolate our ingredients:

  • Replace rendered-body masks with EgoHOS masks.
  • Remove the grasp-velocity coupling.
  • Ground contact against the static background only, which reproduces the static-world contract.

Beating the cascade means lower object pose error on HOI4D frames where hand occlusion exceeds half the object's area, and lower hand and body error during contact. On whole sequences we require the joint fit to be no worse. A gain that appears only on average and not in occluded contact frames would not count.

Why not just…?

Why not just run EgoGaussian and EgoAllo separately and fix penetrations afterward? This is the real threat, and it may capture much of the pose benefit. Its weakness is structural. It cannot pass information from the hand to the object, so object tracking under occlusion inherits EgoGaussian's segmenter dependence. Its background still relies on 2D masks that fail when limbs cross or unfamiliar tools appear. The cascade is our main comparison. If it matches us on occluded-contact object error, the coupling claim is refuted, and we would retreat to the narrower claim about masks.

Why not just mask the wearer with SAM or EgoHOS and reconstruct a static scene? Evidence from RobustSplat and Split4D shows that mask errors and missed frames become permanent 3D artifacts. A static model also cannot represent moved objects at all. The naive baseline quantifies this directly. The rendered-mask ablation tells us whether our own mask is better than EgoHOS, or merely different.

Why not just use an unconstrained deforming Gaussian model for everything, body included? EgoGaussian's HOI4D comparison already shows that unconstrained 4DGS is about 5 dB worse on dynamic frames than rigid objects. Free deformation also produces no joint angles, so it cannot be scored on pose or contact.

Why not just train a feed-forward network on photorealistic synthetic renders? The file's evidence is discouraging: BundleTrack drops to 19.3% on real HOI4D, and FPN reaches only 21.36% box AP on ADT. We test this directly rather than assume it. We run EgoMono4D, a self-supervised feed-forward 4D reconstructor, on both synthetic and real ADT inputs. We also measure its paired gap against ours.

Proposed Experiments

On HOI4D we evaluate object nine-degree-of-freedom tracking accuracy stratified by hand-occlusion fraction, MANO hand error, and dynamic-frame PSNR. EgoGaussian's reported 30.33 dB is the number to beat.

On ADT we evaluate:

  • body MPJPE;
  • object six-degree-of-freedom error, with ADT's ground truth accurate to 6.83 mm and 1.29°;
  • penetration rate against ground-truth dynamic object poses;
  • depth error.

The decisive synthetic-to-real check runs every method on both the real stream and the pixel-aligned synthetic render of the same ADT sequence and reports the error difference. Geometry and motion are identical between the two, so the difference isolates the effect of appearance and sensor artifacts. We predict that our method's difference is smaller than EgoMono4D's and EgoGaussian's. The synthetic renders also supply amodal background behind the wearer's hand. We use them to score PSNR on the background region under the hand, which no real benchmark can provide.

On Ego-Exo4D we render the reconstructed wearer and scene from the calibrated third-person GoPro viewpoints and compare against those frames. This tests whether the body layer is geometrically right and not only plausible from the head's own viewpoint.

The proposal fails if any of the following holds:

  • The cascade matches it on occluded-contact frames.
  • Rendered-body masks are no better than EgoHOS.
  • Its synthetic-to-real difference on ADT is as large as that of the feed-forward model.