Seeing Where a Hidden Joint Cannot Be: Ray-Searched Stereo Attention for Egocentric Pose

Headsets and smart glasses that track the wearer's own body use two downward-looking fisheye cameras. The wearer's torso, arms, and the edges of the image keep hiding the legs and feet. Current systems fill in these hidden joints from learned body priors and treat "I cannot see the foot" as having no information. That is wasteful. When the foot is hidden behind the torso, stereo still tells you how far away the torso is along the line of sight, so the foot must lie behind that surface. When the foot falls outside the field of view, it must lie outside the camera's viewing cone. We propose to turn this negative evidence into an explicit geometric constraint, computed by a cheap stereo search along each camera ray.

Concretely, each joint does three things in the stereo attention step. It samples a handful of 3D points along the camera ray through its current estimate. It scores how well the two fisheye views agree at each point, which reveals where a real surface is. It then decides whether that first surface is the joint itself or something blocking it. If the joint is visible, the search pins down its depth. If it is hidden, the search returns a depth below which the joint cannot lie, and the pose refiner is penalized for placing the joint in front of it. The aim is to cut error on hidden and distal joints without adding cameras, video, or offline scene reconstruction.

Core Claim

Existing egocentric stereo methods differ mainly in how they fuse the two views. None of them treats self-occlusion as a measurable geometric event.

EgoPoseFormer works in two stages:

  • A small network proposes a coarse 3D skeleton.
  • Each joint, represented as a query token, is projected into both views and refined with deformable attention, meaning it samples a few learned offsets around the projected point.

When the proposal's depth is badly wrong, which is common for hidden limbs, the samples sit around the wrong image location. Hidden joints are then filled in purely by self-attention among joint tokens.

Ego3DPose matches cropped limbs across views and reports 60.82 mm mean per-joint position error (MPJPE, the average 3D distance between predicted and true joints) on UnrealEgo. Bring Your Rear Cameras masks low-confidence heatmap regions and adds two rear cameras.

Two non-egocentric works supply the pieces we reuse:

  • Epipolar Transformers samples along the epipolar line, the line in the second image where a point from the first image must appear. Its purpose is to fuse features for outside-in pinhole cameras. It does not reason about occluders.
  • iPASSR shows that cross-view attention becomes flat or empty where a region is visible in only one view. It uses this only to switch fusion off.

Our claim is that, in the egocentric case, the occluder is almost always the wearer's own body or the image boundary. A per-joint search along the ray therefore yields a usable lower bound on hidden-joint depth. Enforcing that bound should reduce error specifically on joints hidden in both views.

This differs from all prior egocentric work in that occlusion becomes a constraint rather than a mask. It differs from epipolar transformers in two ways. First, it searches in 3D along a ray and projects through the fisheye model, so curved epipolar lines never need to be traced. Second, it uses the location of the best match even when that match is not the joint.

Research Program

We build on EgoPoseFormer's coarse-to-fine design and replace its stereo attention with a ray-search module.

Ray sampling. For joint $j$ with current estimate $P_j$ and camera $c$ with optical center $o_c$, we sample $D$ points along the ray:

$$X_{j,d} = o_c + t_d \frac{P_j - o_c}{\lVert P_j - o_c \rVert}, \qquad d = 1, \dots, D.$$

The depths $t_d$ span a fixed near-to-far range around the head. Each point is projected into both views through the calibrated fisheye model $\pi_L, \pi_R$. Because the search is defined in 3D, the 170° distortion is handled by projection alone.

Two scores per sample. At each sample we compute two quantities:

  • A cross-view consistency score $s_{j,d}$, the feature similarity between $\pi_L(X_{j,d})$ and $\pi_R(X_{j,d})$. This acts as a sparse, per-joint plane sweep that says "a surface exists here."
  • An identity score $r_{j,d}$, the similarity between joint $j$'s query and the fused feature. This says "that surface is joint $j$."

What the profile tells us. If the consistency and identity peaks coincide, the joint is visible and its depth is read out softly. If the consistency profile peaks but the identity score stays low there, we take the expected depth of the first strong surface, $\hat t^{\,occ}_{j,c}$, as the occluder depth for that ray. If the projection falls outside the image, the joint is flagged as clipped.

The constraint. The occluder depth and a soft visibility probability $v_{j,c}$, derived from the identity profile, feed back into the joint token. They also define a hinge penalty:

$$\mathcal{L}_{shadow} = \sum_{j,c} (1 - v_{j,c}) \, \max\!\left(0,\; \hat t^{\,occ}_{j,c} - \lVert \hat P_j - o_c \rVert\right).$$

Here $\hat P_j$ is the refined prediction. The penalty is applied at training time, and optionally as a test-time projection step. It is one-sided by design: it only forbids a hidden joint from sitting in front of the surface that hides it, so it cannot pull visible joints off their evidence. For clipped joints, an analogous term keeps the projected prediction outside the image.

What gets tested. Three questions:

  • Does ray search recover from bad proposals better than local deformable sampling at the same number of samples?
  • Does the shadow constraint help hidden joints beyond what ray search alone provides?
  • Does visibility inferred from attention work nearly as well as visibility labels derived from ground truth?

Baselines

Naive stereo reference. The UnrealEgo baseline concatenates early convolutional features from the two views and lifts 2D heatmaps to 3D. It reports 79.06 mm MPJPE on UnrealEgo. We cite this number and also rerun the model with our data pipeline. Its role is to establish how much any explicit stereo reasoning buys.

Strongest frame-based competitor. This is EgoPoseFormer, which reports a 27.4 mm (45%) MPJPE reduction over Ego3DPose's 60.82 mm on UnrealEgo. It is the strongest competitor because it is the most accurate frame-based stereo model in the file and the lightest, using 7.9% of Ego3DPose's parameters. Our method is a drop-in change to its stereo attention.

To keep the comparison fair, we retrain it with:

  • the same backbone (ResNet-18 and ResNet-101),
  • the same training schedule,
  • the same number of sampled points per joint per view,
  • the same hyperparameter search budget.

Beating it means a lower MPJPE on joints hidden in both views, with a 95% bootstrap interval excluding zero, and no regression on visible joints, at matched FLOPs.

Ceilings rather than targets. - 3D Human Pose Perception from Egocentric Stereo Videos reports 30.53 mm on UnrealEgo2, but uses video and offline structure-from-motion. We report it as a ceiling, not a like-for-like target. - On Ego4View-RW, adding rear cameras improves EgoPoseFormer's MPJPE by 18.7%. We report how much of that gain our method recovers using only the front pair.

Ablations isolating the key ingredients. The ablation that isolates the central idea is ray search without $\mathcal{L}_{shadow}$ and without occluder-depth feedback. This separates the value of searching along depth from the value of using negative evidence. A second ablation replaces inferred visibility with labels derived from ground truth.

Why not just…?

Why not just widen EgoPoseFormer's deformable offsets or add more refinement layers? This is the most serious threat. Learned offsets could in principle spread along the depth direction on their own. We test it directly with three variants at an identical sample budget:

  • deformable attention with enlarged initial offset range,
  • deformable attention with additional refinement iterations,
  • our ray search.

We also inject controlled depth errors into the coarse proposals. If widened deformable sampling degrades as gracefully as ray search, the sampling half of our proposal reduces to a tuning change. In that case the contribution rests entirely on the shadow constraint, which deformable attention does not provide.

Why not just use iPASSR-style or heatmap-uncertainty masking? Masking, as in iPASSR and the rear-camera paper, tells the model to ignore unreliable evidence. It still leaves the hidden joint to the prior with no geometric bound. We compare against an entropy-masking variant of our module that uses the same ray profiles but no hinge. Any gap on doubly hidden joints is attributable to using the occluder depth rather than discarding it.

Why not just add temporal context or a motion prior? Video helps, as the drop to 30.53 mm on UnrealEgo2 shows. However, that result relies on offline structure-from-motion, and EgoEgo reports 121.1 mm on ARES from head motion and a diffusion prior alone, so priors without metric grounding remain loose. The shadow constraint is per-frame and composes with temporal models rather than competing with them. We will additionally report our module plugged under a simple temporal smoother, to check that its gains do not vanish once temporal context is present.

Why not just add rear cameras? Rear cameras work, giving the 18.7% gain above. They also add payload and power, and are blocked by hair. The frontal-only comparison on Ego4View-RW quantifies how much of that gain is recoverable in software. We expect a partial recovery, not parity.

Proposed Experiments

Training and in-distribution evaluation. We train on UnrealEgo and UnrealEgo2 and evaluate on their test sets. We report:

  • MPJPE and PA-MPJPE (error after rigidly aligning prediction to ground truth),
  • per-joint error from head to toes,
  • error split by visibility: visible in both views, hidden in one, hidden in both, clipped by the image boundary.

Visibility strata are computed by rendering the released body models against ground-truth joints where the assets permit. Where they do not, we fall back to a frustum-only proxy and state that the stratification is coarser.

Real-world transfer. We fine-tune and test on UnrealEgo-RW and on the front views of Ego4View-RW. For UnrealEgo-RW, the file gives only a relative 10% improvement for the video method and no absolute MPJPE, so we rerun the baselines ourselves.

Decisive checks. There are three.

  1. Proposal-noise sweep: add depth perturbations of increasing size to the coarse proposals and plot hidden-joint error versus perturbation for deformable attention and ray search.
  2. Shadow-constraint ablation, restricted to joints hidden in both views. This is where Epipolar Transformers-style fusion is, by its own logic, blind, and where the constraint should matter most.
  3. Calibration-perturbation test: perturb extrinsics slightly to check that a 3D ray search is not more fragile than local sampling.

Efficiency. We report FLOPs and measured latency against EgoPoseFormer and Ego3DPose's reported 18 ms, since a wearable-scale budget is part of the claim.

Falsification. The proposal fails if hidden-joint error does not improve over the masking variant. It also fails if inferred visibility is so unreliable that the hinge helps only with ground-truth-derived labels.