Reading Silence: Egocentric Event-Camera Forecasting That Treats Missing Events as Evidence

An event camera is a sensor whose pixels report only changes in brightness, not full images. It is attractive for headsets because it does not blur during fast head turns and keeps working in the dark. But it goes quiet whenever nothing moves. For a downward-looking headset camera aimed at the wearer's own body, that quiet is ambiguous. A leg can be missing from the stream for two different reasons. It may be hidden behind the torso, in which case we do not know where it is. Or it may be in plain view but motionless, in which case we know almost exactly where it is and only not when it will move again. Current systems treat both cases as missing data and bridge the gap with a memory buffer that slowly decays. That is wrong in both directions, and most wrong at the moments that matter for prediction: a person standing still at a doorway, about to choose a direction.

We propose a forecaster of future full-body motion from a head-mounted event camera that explicitly decides why each body part is silent. It uses the camera's known geometry and the current pose estimate to tell "hidden by my own body" apart from "visible but still." When silence means stillness, it confirms the body's state instead of letting it decay. When silence means occlusion, it widens uncertainty. The output is a set of sampled future trajectories, not a single average. The claim to test is simple. Explaining away silence should (a) stop the pose drift that existing event-based memory modules show during long pauses, and (b) produce better-calibrated predictions of how and when a paused person will set off again.

Core Claim

Three bodies of work meet here, each in a different role.

Static scene blindness is the core mechanism. Event pipelines lose tracks and degrade when motion stops, as shown for object tracking in Overcoming Static Scene Blindness in Event Vision and Dietsche et al.. Two families of fixes exist. Memory modules include the Residual Event Propagation Module (a feature buffer that carries past joint and body-mask features forward) in EventEgo3D and EventEgo3D++, plus the gated long-range memory in DMANet. Hardware fixes jiggle the optics, such as AMI-EV and VibES. These works motivate the proposal: the memory approaches are reported to drift during extended stillness. They also constrain it, because they treat silence as absence of information.

Egocentric self-occlusion is what makes silence ambiguous. The torso blocks the lower body in downward headset views (xR-EgoPose), and the feet sit at the low-resolution edge of the fisheye image. In a third-person event setup like YeLan, a silent torso is almost always a still torso. On a headset it often is not.

Multi-modal trajectory prediction supplies the output. That means predicting a distribution over several distinct futures rather than one averaged path, which avoids the "averaging trap" documented by Multiple Futures Prediction.

The obvious combination would be EventEgo3D++'s encoder plus a diffusion forecasting head. Nobody has published it; the closest systems are EgoCast, which forecasts egocentric pose but from RGB with a deterministic regression head, and E-Motion, which generates future event video but only from a third-person view. We include that combination as a baseline rather than as our contribution.

Our contribution is a per-joint negative-evidence observation model. Let $s$ be the current body state. Let $n_j$ be the number of events in the image patch around joint $j$ over a short window. Let $v_j \in [0,1]$ be the probability that joint $j$ is geometrically visible, rendered from the estimated skeleton and the calibrated fisheye rig. Let $\lambda_j(s)$ be the expected event count for that patch given the joint's motion relative to the camera. Then the probability of observing silence is

$$p(n_j = 0 \mid s) = (1 - v_j) + v_j \, e^{-\lambda_j(s)}.$$

The first term covers occlusion. The second covers a visible patch that emits nothing, approximating event emission as a Poisson process whose rate grows with relative image motion. When $v_j$ is high, silence pushes the state toward low relative velocity and pins the joint in place. When $v_j$ is low, silence is uninformative and the forecaster must lean on its learned motion prior and the head trajectory, the head-to-body bridge used by EgoEgo.

Research Program

The system has three parts.

Encoder and nowcaster. We start from EventEgo3D++'s lightweight encoder (1.25M parameters, running at 140 Hz on a mobile GPU). We replace its representation of events as locally normalized 2D surfaces with one that also keeps time-since-last-event, so the network sees recency directly.

Belief module. A small recurrent state-space module maintains the body and root state. At each step it applies the silence likelihood above. Visibility $v_j$ is rendered differentiably from the current skeleton. The expected rate $\lambda_j$ is predicted by a small learned head from the joint's projected velocity and the local edge contrast. That contrast is cached from the most recent moment the patch was active, so it remains available during stillness. The module also takes head rotation from the event stream and headset IMU. This lets it account for the "background storm," the flood of background events during head turns, rather than reading it as body motion.

Forecaster. A conditional diffusion model generates future root paths and joint trajectories over a horizon of up to 5 seconds. A diffusion model is a generator that learns to turn random noise into samples step by step. This one is conditioned on the belief state. Following EgoNav, we use DDIM/DDPM hybrid sampling to produce around 15 samples in real time. Following Trajectron++, the root is generated as velocity and heading controls integrated through a simple walking-dynamics model, which suppresses the long-horizon jitter that EgoCast reports.

Data. No egocentric event forecasting benchmark exists, so we repurpose the EventEgo3D datasets: EE3D-S (synthetic, 946 sequences), EE3D-R (studio), and EE3D-W (in the wild). We cut their sequences into past/future windows. EE3D-S is the main training set. We also render a pause-heavy synthetic split: AMASS motions run through the same downward-fisheye event simulation pipeline, selected for stand-still-then-depart segments with ground-truth departure times and directions. Each test window is stratified by pause length and by the visibility of the paused joints, so the occluded-silent and visible-silent cases can be scored separately.

Baselines

Do nothing. A constant-pose, zero-velocity extrapolation of the last nowcast pose. During pauses this is a hard reference, because "stay put" is right until departure. Any method must beat it on departure-timing and post-departure error, not just mean error.

Strongest existing system, as a two-stage pipeline. EventEgo3D++ nowcasting, which reports 98.67 mm mean per-joint position error (MPJPE) on EE3D-S as a pose estimator, feeds its pose history into an EgoEgo-style conditional diffusion forecaster trained on AMASS. This is the strongest competitor because it combines the best event-based egocentric pose estimator with an established multi-modal body-motion prior. Its weakness is that pause drift enters the forecast unexamined.

Obvious end-to-end combination. The same encoder, memory buffer, and diffusion head, trained jointly but without the silence likelihood.

EgoCast-style RGB forecaster. Run where an RGB stream is available, mainly synthetic renders. This tests the claim that event input matters under fast motion and low light. EgoCast reports an MPJPE area-under-curve of 24.41 cm on Ego-Exo4D over a 5 s horizon. That is a different sensor and dataset, so it is context only. No reported result was found on Lacuna for any forecasting method on EE3D-S, EE3D-R, or EE3D-W, so every number on those splits will be new.

Fairness. All learned baselines share the same backbone size, training data, number of diffusion samples, compute budget, and hyperparameter search budget, which we fix at an equal number of trials.

What counts as beating them. We need lower best-of-K and sample-quality error, especially on windows with pauses over 2 seconds and on visible-silent joints. We also need no regression on moving segments.

Why not just…?

Why not just keep a longer memory buffer, or feed time-since-last-event into the network and let it learn? This is the most serious threat. YeLan's recency volume already encodes silence, and a network given that plus enough data might learn the occluded-versus-still distinction on its own. We test it directly with an ablation ladder:

  • recency input alone;
  • recency input plus the rendered visibility mask as an extra channel;
  • the full explicit silence likelihood.

If the second rung matches the third, the real contribution is the self-occlusion mask, not the probabilistic model, and we will report it that way. We expect the explicit model to win mainly on pauses longer than anything common in training, where learned implicit memory has no reason to stay stable.

Why not just add an actuator (AMI-EV, VibES) so the sensor never goes silent? These fixes make static scenes visible. They do not fix occlusion: a vibrating sensor still cannot see a foot behind the torso. They also add weight and moving parts to a rig that EventEgo3D already reports at 0.42 kg. We do not test hardware, and we acknowledge that on a static tripod an actuator would likely remove the stillness half of the problem.

Why not just extrapolate from the head trajectory with an IMU? EgoEgo shows that head motion maps to many possible body motions. EgoCast reports that adding visual context cuts ankle error by 39% relative to head-only input, which shows head-only models struggle with the lower body. Our IMU-plus-head-only diffusion ablation measures how much the event stream adds, separately during pauses and during motion.

Proposed Experiments

Metrics. - Best-of-15 and mean joint MPJPE, plus root displacement error at 1, 3 and 5 seconds. - Miss rate for final root position. - A calibration score on samples (energy score) to catch over-dispersed distributions. - Departure-time error after pauses. - Foot-skating and penetration rates.

Decisive check 1: error versus pause length. We plot error against pause duration. Memory-buffer baselines should drift with pause length, while the silence-aware model should stay roughly flat for visible joints.

Decisive check 2: counterfactual event masking. On held-out windows we delete events from a visibly moving limb, which should make the model infer occlusion or stillness depending on the rendered $v_j$. We also inject synthetic occlusion. A model that truly reads silence should widen its distribution only in the occluded case.

Decisive check 3: sim-to-real transfer. Training on EE3D-S and testing on EE3D-R and EE3D-W shows whether the learned expected-rate head survives real sensor noise and the background storms of head turns.

What would falsify the idea. If the end-to-end baseline with the visibility channel matches the full model on long pauses and on the masking test, the explicit negative-evidence model is unnecessary.