Reading Event Silence: Visibility-Aware Stability Buffering for Egocentric Event-Camera Pose

An event camera is a sensor in which each pixel fires independently, reporting a timestamped "event" whenever its brightness changes, instead of capturing whole frames at a fixed rate. Head-mounted event cameras can already track the wearer's body at 140 Hz without motion blur (EventEgo3D, EventEgo3D++). They have one characteristic failure. When a limb produces no events, the tracker cannot tell why, and its memory drifts. A leg that stands still in plain view is silent. A leg hidden behind the wearer's own torso is also silent, as is a hand that has left the field of view. These three cases call for opposite responses: hold the pose firmly, let it relax toward a body-motion prior while uncertainty grows, or give up on the joint. Current systems treat all three the same way, by carrying forward a learned feature buffer.

We propose to treat silence as a measurement rather than as missing data. The camera geometry is fixed and the tracker keeps a running body estimate, so the system can render which joints the torso currently hides. For a joint it judges visible, a lack of events is positive evidence that the joint is not moving relative to the head, because any motion would have fired pixels. How much evidence the silence carries depends on how much texture and contrast that limb shows, and the system can measure this from the events the limb produced the last time it moved. This turns the fixed dwell-time thresholds used in interface design into an adaptive statistical test. The per-joint wait before the system concludes "still" becomes longer for a low-contrast trouser leg and shorter for a high-contrast sleeve.

Core Claim

Two existing strategies bridge event silence. Recurrent feature buffers propagate past states (EventEgo3D's Residual Event Propagation Module). Attention over past frames serves the same purpose in RGB video (Ego-STAN). Our claim is that a per-joint filter that explicitly separates "silent and visible" from "silent and occluded" will beat both on the conditions where they are known to fail. Those conditions are long stationary holds and prolonged torso self-occlusion. The filter adds only a small cost at inference, and it should not lose accuracy on ordinary moving segments.

The key quantity follows from the standard event-generation model, in which a pixel fires when its log-brightness changes by a contrast threshold. Suppose limb $j$ moves at image speed $v$. It then produces events at an expected rate $\lambda_j(v) \approx \rho_j v + \eta$, where $\rho_j$ is the limb's events per pixel of motion (estimated online from its recent moving bursts) and $\eta$ is the sensor noise rate. Under a Poisson approximation, the probability of seeing no events over a window $\Delta t$ while the limb moves at least a minimal speed $v_{\min}$ is $\exp(-\lambda_j(v_{\min})\Delta t)$. Declaring stillness at false-alarm level $\alpha$ therefore requires a dwell time

$$\Delta t_j^{*} = \frac{\ln(1/\alpha)}{\rho_j v_{\min}},$$

which is the adaptive replacement for a fixed threshold such as the 120 ms gaze dwell in EmBARDiment or the 10 s reading threshold in Neural News Recommendation with Negative Feedback. Those works motivate the sustained-evidence principle. They also illustrate the weakness we address: a single global threshold ignores how informative the signal actually is.

Research Program

We build a lightweight per-joint filter on top of the EventEgo3D++ backbone. Each joint carries a 3D position, a velocity, and an uncertainty. At every update step, the current skeleton estimate is posed with a coarse body mesh and rendered through the calibrated fisheye model, which gives a visibility probability for each joint. That probability is the self-occlusion reasoning highlighted as the central difficulty in xR-EgoPose and UnrealEgo.

The filter then applies one of three measurement modes. When the joint is visible and emitting events, the network's joint prediction is fused as a normal measurement. When the joint is visible but silent, a zero-relative-velocity measurement is applied. Its strength grows with the accumulated log-likelihood ratio $\rho_j v_{\min}\Delta t$, so the filter commits to stillness gradually, with hysteresis, rather than snapping. When the joint is occluded, no image measurement is applied. The state is propagated with a kinematic prior, and uncertainty grows at a rate learned from data. For the prior, we use a head-trajectory-conditioned body model in the spirit of EgoEgo, with bone-length constraints from EventEgo3D++. The file documents that head-only priors are one-to-many; here the prior only fills gaps rather than driving the whole estimate.

A second cue marks the transitions between modes. Following ShapeAug, which showed that moving occluders generate event trails along their silhouettes, we expect a burst of events along the rendered torso boundary as a limb disappears behind it. We use that burst to confirm an entry into occlusion instead of mistaking it for a sudden freeze. The whole filter is differentiable, so the uncertainty-growth rates and $v_{\min}$ can be tuned end to end against ground truth while $\rho_j$ stays estimated online.

Baselines

The naive reference is EventEgo3D++ run frame by frame with its temporal buffer disabled. This shows how much any temporal memory helps on each condition.

The strongest existing method is the full EventEgo3D++. It reports 98.67 mm MPJPE (mean per-joint position error) on the synthetic EE3D-S benchmark with 1.25 million parameters at 140 Hz. For the real-studio EE3D-R set, no reported MPJPE was found on Lacuna, so we will retrain and report it ourselves. We will retrain from the released architecture on identical splits with the same training budget, and give our filter the same backbone weights. Beating it means lower MPJPE in the still-visible and long-occlusion strata, with no worse than parity (within seed variance) on moving-visible segments and overall EE3D-S error below the 98.67 mm figure.

The first ablation isolates the key ingredient by replacing visibility-gated modes with a visibility-blind zero-velocity update. This baseline freezes any joint whose event count drops below a threshold. Its threshold is tuned on validation data with the same search budget as our parameters.

The second ablation keeps visibility gating but uses a fixed dwell time instead of the contrast-adaptive $\Delta t_j^{*}$. It tests whether the adaptive threshold matters beyond the three-way split.

The strongest simple learned competitor feeds the rendered visibility map as an extra input channel to EventEgo3D++'s recurrent buffer and retrains the whole network. This tells us whether an explicit filter is needed at all, or whether visibility information alone is enough.

Why not just…?

Why not just let the recurrent buffer learn this implicitly with more occlusion augmentation? EventEgo3D++ already trains on large synthetic data, yet its authors report drift when events become sparse or limbs are occluded for long windows. That result suggests implicit learning does not pick up the distinction on its own. The visibility-as-input baseline is the real threat here. If it matches our filter, the honest conclusion is that the contribution is the visibility signal itself, not the filtering machinery. We will report it that way.

Why not just use a Kalman filter with zero-velocity updates whenever events go quiet? This is the closest simple idea, and it is exactly our first ablation. Its predicted failure is systematic: it freezes occluded legs in place at the moment they disappear behind the torso. It also locks onto slowly moving low-contrast limbs. MATE documents the related failure of constant-velocity bridging under non-linear motion during event "blind time." The occluded stratum and the low-contrast still stratum will show whether this failure appears.

Why not just add low-rate RGB frames as anchors, as EventCap does? An RGB anchor frame would largely solve static blindness for visible limbs, so this is a genuine competitor on the stillness half of the problem. It does nothing for joints hidden behind the torso, however, and it adds a second sensor stream to the headset. If a 5 Hz RGB anchor matches us on still-visible segments, which we will test wherever rendered RGB can be paired with EE3D-S events, our contribution narrows to the event-only setting and to occlusion handling.

Why not just infer occluded limbs from head motion alone? The file notes that EgoEgo reaches 121.1 mm MPJPE on ARES, which is worse than direct event observation, and that the head-to-body mapping is one-to-many. We use such a prior only inside the occluded mode, and we will report head-prior-only tracking as an additional reference.

Proposed Experiments

We evaluate on EE3D-S, EE3D-R, and EE3D-W. Per-joint visibility labels come from rendering ground-truth meshes through each dataset's fisheye calibration. These are synthetic bodies for EE3D-S, motion-capture fits for EE3D-R, and the provided SMPL body-model fits for EE3D-W. Every frame and joint is then labelled moving-visible, still-visible, or occluded, and silent and occluded segments are further binned by duration.

To isolate stillness, we will also generate controlled sequences with the Differentiable Event Stream Simulator approach. In these sequences, AMASS-style motions are paused for 0.1 to 5 seconds and limb textures range from high to low contrast. This design directly tests the prediction that the required dwell time scales inversely with $\rho_j$.

Metrics are MPJPE and PA-MPJPE (MPJPE after aligning the predicted skeleton to ground truth) per stratum, drift slope in millimetres per second of silence, acceleration error as a jitter measure, and latency to re-lock after a joint reappears. We add the 1.25 million-parameter backbone's runtime and measure the filter's overhead on the same mobile-class GPU.

The decisive check has two parts. On still-visible segments longer than one second, our method's drift should stay near zero while EventEgo3D++'s drift grows with duration. On occluded segments, the visibility-blind zero-velocity ablation should show the largest error of all methods, and our method should not. If both patterns hold and the fixed-dwell ablation falls between the two, each ingredient has earned its place. If instead the visibility-input baseline matches us everywhere, we will conclude that the explicit filter is unnecessary.