Remembered Ground: Local Gaussian Maps for Grounding Self-Occluded Egocentric Pose

A camera on a headset or a pair of smart glasses rarely sees the wearer's own feet. The torso blocks the view from a downward-looking camera, and a forward-looking camera sees the wearer's limbs in under 10% of daily activities. Pose estimators therefore guess the legs from almost nothing, and the guessed bodies float above the floor, slide across it, or sink into furniture. The one method in our file that grounds egocentric pose in the surrounding scene, Scene-aware Egocentric 3D Human Pose Estimation, handles this by predicting, from a single frame, what the depth of the floor probably is behind the body. It then reasons about contact in a dense voxel grid, which is slow. Our observation is simpler. The floor under the wearer's feet right now was usually visible a second or two ago, before the wearer stepped onto it. The camera has already seen the support surface. It just has not kept it.

We propose to keep it. The system maintains a small, bounded 3D map of the last few seconds of surroundings. The map is made of 3D Gaussians, which are explicit colored blobs that a graphics card can render at hundreds of frames per second. The pose estimator queries this map around each predicted joint, so occluded limbs are grounded in geometry that was actually observed, not hallucinated. The predicted body also feeds back into the map. Its silhouette tells the mapper which pixels belong to the wearer, so the wearer is not baked into the background as floating artifacts. The main claim can be tested directly: remembered local geometry should reduce error on distal joints and penetration. It should help most on frames where the support surface was seen earlier, and it should not help on frames where the surface was never seen.

Core Claim and How It Differs

Three bodies of prior work each hold one piece of this. Egocentric pose estimators ignore the scene entirely. xR-EgoPose encodes joint uncertainty with a dual-branch heatmap autoencoder, UnrealEgo uses stereo fisheye disparity, and EgoPoseFormer refines coarse joint proposals with sparse deformable attention. All three process single frames, and none uses the environment. EgoEgo generates the body from head trajectory with a diffusion model and explicitly reports floor and furniture penetration. The scene-aware method is our closest competitor and the main motivation. It shows that scene geometry matters, but its geometry is inpainted per frame and processed in voxels, with no memory across time.

Dynamic-scene reconstruction systems do the opposite: they treat the wearer as noise to remove. OmniLocalRF inpaints the photographer out of local radiance blocks. DeGauss and DG-SLAM separate moving people from a clean static map. EgoGaussian masks hands out when it builds the background. None of these estimates the wearer's posture.

Local-domain mapping supplies the third piece, a bounded map. OmniLocalRF and the sliding-keyframe design of DenseSplat show that a map windowed in space and time keeps memory bounded without global structure-from-motion. DenseSplat reports 6.67 GB, against 11–18 GB for its baselines. We reuse that windowing idea.

What is new is the coupling and the specific role memory plays in it. The map exists to answer one question cheaply: "what surface is near this occluded joint?" The body exists to tell the map which pixels are self.

Research Program

The map is a sliding window of 3D Gaussians fitted to the last few seconds of keyframes, expressed relative to the current head pose. Keyframes older than the window are evicted, so memory stays constant regardless of how long the trajectory is. Camera poses come from the dataset when it provides them; otherwise we estimate them with visual odometry, following EgoEgo's use of DROID-SLAM. Monocular fisheye input gives weak depth, so the map uses the datasets' depth or a monocular depth prior during fitting. This is a real risk, and we quantify it below.

The pose network follows EgoPoseFormer's two-stage design. A coarse stage proposes joint locations. Before refinement, we render the local map into a small downward-looking depth patch beneath each proposed joint and around it. This takes K tiny renders, far cheaper than a full frame. Each patch is encoded into a scene token that the refinement transformer attends to alongside its image features. Training adds two terms adopted from the scene-aware method. A penetration penalty pushes joints out of the surfaces the map renders, and a contact term pulls low-velocity feet toward the nearest rendered surface.

The feedback path works as follows. The refined skeleton is rasterized as a coarse capsule body into the fisheye image. The pixels it covers, dilated slightly, are excluded from the map's photometric loss. Gaussian splitting is also disabled inside them. This targets the failure documented by RobustSplat: moving objects trigger aggressive densification and leave floaters behind. RobustSplat's fix is to delay densification globally, which suits offline photo collections. Here the body tells us where to withhold densification, while the rest of the map keeps updating online.

Baselines

The naive reference point is EgoPoseFormer retrained on each target dataset with no scene input. This measures how much the scene adds at all. The file gives no absolute MPJPE for EgoPoseFormer, only relative gains: 27.4 mm on UnrealEgo and 25.5 mm on SceneEgo over prior methods. We therefore retrain it ourselves under the same splits and schedule.

The strongest existing method is Scene-aware Egocentric 3D Human Pose Estimation. It reports MPJPE falling from 189.6 mm (the EgoPW baseline) to 118.5 mm, contact rate rising from 38.8% to 89.4%, and a non-penetration rate of 84.1%. It is the only method that already uses scene geometry for this exact problem. We run it with its released training recipe on EgoGTA and EgoPW-Scene and evaluate on its real-world interaction test set. Both systems get the same frames. Our system additionally sees past frames, which is precisely the variable under study. Beating it means lower MPJPE and higher contact and non-penetration rates on its own test set, with per-frame latency in the tens of milliseconds rather than the cost of a voxel network.

The baseline that worries us most is a decoupled pipeline. Run EgoPoseFormer, build the same local Gaussian map using off-the-shelf 2D body segmentation masks, then fix the predicted pose by a short post-hoc optimization against the map's penetration and contact terms. This uses every ingredient except learned coupling. It gets the same map window, the same compute budget for the optimization as our refinement stage, and hyperparameters tuned on the same validation split.

Two ablations isolate the key ingredients. The first, "no memory," builds the map only from the current frame. This is our analogue of per-frame inpainting, and it isolates the value of remembering. The second, "segmenter masks," replaces body-derived masks with the 2D segmenter. It isolates the feedback path, and we measure it by floater count and background PSNR in the map.

Why not just…?

Why not just mask the wearer with a 2D segmenter and run static Gaussian splatting plus a separate pose estimator? This is the decoupled pipeline above. It is a serious threat for pose accuracy, because if the map is good, post-hoc optimization may recover most of the gain. Evidence against it is partial. Split4D reports that 2D video segmenters drift under fast motion and contact, and DG-SLAM had to add geometric depth-warp masks because semantic masks alone missed dynamic content. The experiments settle the question two ways. If the pipeline matches us on MPJPE, our contribution reduces to "use a remembered map," which is still the main claim. The learned coupling must then justify itself through cleaner maps and lower latency, or be dropped.

Why not just condition a motion generator on head pose, as EgoEgo does? EgoEgo reports 121.1 mm MPJPE on its ARES dataset, but it acknowledges that its bodies clip floors and furniture because it has no scene. Adding our scene tokens to a head-conditioned generator is possible later work. As a competitor, it lacks exactly the grounding we test, and we evaluate it on non-penetration on ARES, whose Replica scenes provide geometry.

Why not just inpaint scene depth per frame and use voxels, as the scene-aware method does? Inpainting guesses surfaces that our map has actually observed, and voxel 3D convolutions spend most of their compute on empty space. EgoPoseFormer reached its gains with 13.1% of the FLOPs of a volumetric baseline. The "no memory" ablation tests the first point directly, and latency measurements test the second.

Why not just keep one global Gaussian map? For short indoor clips, a global map may work just as well, and we will report it. The local window matters for memory and drift on long trajectories, so we compare peak VRAM and accuracy as clip length grows.

Decisive Checks

We train on EgoGTA, which is synthetic with ground-truth depth, and fine-tune on EgoPW-Scene. We test on the scene-aware method's real-world interaction set and on ARES. Metrics are MPJPE, PA-MPJPE, lower-limb MPJPE, contact rate, non-penetration rate, end-to-end latency per frame, and peak map memory. Map quality is measured as background PSNR and floater count on held-out views.

The decisive analysis stratifies test frames by whether the support surface under the feet appeared unoccluded in any frame within the map window. Our hypothesis predicts that gains over the no-memory ablation and over the scene-aware baseline concentrate in the "seen earlier" stratum and vanish in the "never seen" stratum. A uniform gain across both strata would mean the improvement comes from something other than memory. We also vary the window length from zero to several seconds and expect lower-limb error to fall and then plateau. Finally, we degrade depth quality during map fitting to find the point where a noisy remembered map stops beating an inpainted guess.