Holes in the Background Storm: Contact Graphs for Egocentric Event-Camera Hand–Object Tracking

An event camera is a sensor whose pixels each report only changes in brightness, with microsecond timing. That makes it attractive for tracking fast hand–object manipulation that ordinary cameras blur. Mounted on a moving head, though, the sensor sees every static edge in the room sweep across it, and nearly every pixel fires. Prior work treats this "background storm" as noise to be filtered out. Filtering creates its own failure. When a user holds a cup still and turns their head, the hand and cup move with the camera and produce no events. They vanish just as the background lights up, so any filter that keeps only "independent motion" deletes exactly the objects we want to track. We propose to read the storm as signal instead. A hand or held object that moves with the camera blocks the background behind it, so it shows up as a silent hole in the storm with events bunched along its outline. Which entities move together is set by physical contact: a held object moves with the hand, and a supported object moves with the world. So we explain the whole event stream with a small contact graph. Its nodes are camera, world, hands, and objects. Its edges are holding or supported_by relations that force the connected entities to share one rigid motion. An edge is allowed only if the corresponding 3D meshes actually touch, without floating and without interpenetrating beyond skin compliance.

The two-sentence version is this. Each event is assigned to the entity whose motion explains it, and contact edges in a scene graph cut the number of independent motions the tracker has to search over. Mesh contact and non-penetration constraints decide which edges exist, so the graph, the event segmentation, and the 3D hand and object meshes are estimated together, and the background storm becomes evidence about where co-moving foreground objects are.

Core Claim

Under head motion, the correct unit for separating events is the contact-connected rigid group, not the pixel and not the single object. If we estimate which entities are rigidly coupled through physically plausible contact, two things should follow. Hand–object pose should stay accurate through stretches where the foreground produces no events of its own. Penetration and floating should also be lower than in pipelines that denoise events first and enforce physics afterward. The new element is the loop between physics and segmentation. Mesh plausibility gates the scene graph's edges, the edges fix how motion hypotheses are shared, and those hypotheses decide which events belong to which entity, including the silent ones.

Where This Sits

The closest work is Ev2Hands. It tracks two interacting hands from a single event camera, using a segmentation branch that separates left hand, right hand, and background events, and a conic-distance intersection loss between the hands. It reports a 4.09% collision rate. Its authors state that it assumes a fixed camera and that camera motion drowns the hand signal. It also has no objects and no relations.

EventEgo3D++ does handle head-mounted ego-motion. It learns a segmentation decoder that separates body events from background, plus a frame buffer that carries the pose forward when limbs stop moving. Its only physics is a skeletal bone-length loss, and the buffer bridges stillness by memory rather than by explaining it.

EvRGBHand falls back on RGB frames when background events overflow. It reconstructs a single hand with no contact reasoning.

On the physics side, CPF and ContactOpt supply the contact model we reuse:

  • CPF's attractive and repulsive virtual springs, together with its anatomically bounded hand model A-MANO.
  • ContactOpt's finding that strict non-penetration destabilizes optimization, which led it to allow up to 2 mm of penetration as skin compliance.

Both work on single static RGB frames with known object CAD models.

On the structure side, PSG-4D predicts time-stamped relation triplets such as hand-grasping-bottle from egocentric RGB-D video. Its relations are symbolic labels that do not constrain geometry. Its mask tubes degrade under fast camera motion, and that is the regime we target. Interaction Networks motivates computing dynamics over graph edges rather than one global state. InterFormer is the nearest precedent for using relational logic to suppress impossible interactions: its co-occurrence loss cut "ghost hand" illusions from 2.19% to 1.55%. That logic lives on 2D masks with a fixed pixel threshold. We replace the threshold with a 3D contact test.

The bottom-up ego-motion filters constrain the design. These include Vasco et al., who use robot joint velocities, and Zhou et al., who cluster events in space–time. The research file notes that kinematic subtraction cannot tell passive background apart from an object carried rigidly with the platform. That is the case our graph is meant to resolve.

Research Program

The state at time $t$ has three parts:

  • a pose $T_i(t)$ relative to the camera for each entity $i$, where hands are A-MANO meshes and objects are template meshes;
  • a set of contact edges $\mathcal{E}(t)$;
  • a soft assignment of each event to one entity.

The world node's motion relative to the camera is initialized from a head-mounted inertial sensor and refined from the background events. Within an edge $(i,j)$, the child's pose is the parent's pose composed with a fixed relative transform. This removes the child's independent motion parameters for as long as the edge holds.

Each event $k$ gets a residual $r_k(T_i)$. It measures how well entity $i$'s motion explains that event: warp the event along that motion and score how sharply it lands on the entity's projected edges. Entities that move with the camera produce no events of their own. Instead, the predicted occlusion boundary of their mesh against the moving background must match the outline of the silent region. We write that occlusion residual as $o(T_i)$.

The joint objective is

$$ \min_{\{T_i\},\,\mathcal{E}} \; \sum_k \min_i r_k(T_i) \;+\; \alpha \sum_i o(T_i) \;+\; \lambda \sum_{(i,j)\in\mathcal{E}} \big[\phi_{\text{pen}}(M_i,M_j) + \phi_{\text{gap}}(M_i,M_j)\big] \;+\; \lambda \sum_{(i,j)\notin\mathcal{E}} \phi_{\text{pen}}(M_i,M_j), $$

where the terms are defined as follows:

  • $M_i$ is entity $i$'s posed mesh.
  • $\phi_{\text{pen}}$ penalizes penetration deeper than a compliance margin $\delta$. Following ContactOpt, we set $\delta = 2$ mm.
  • $\phi_{\text{gap}}$ penalizes separation larger than $\delta$. It applies only across claimed contact edges.
  • $\alpha$ and $\lambda$ are weights.

Adding an edge therefore has a cost and a benefit. It is cheap only when the meshes really touch, and it pays off because one shared motion then explains the events of both entities. This is the trade that lets physics decide segmentation. Non-contacting pairs are still kept from interpenetrating.

We do not solve this from scratch each frame. A feed-forward network does the first pass. It is a point-cloud backbone in the style of Ev2Hands, run over short event windows, and it proposes entity masks, initial poses, and edge probabilities. A few iterations of the objective then refine the proposal, and the result seeds the next window. This addresses the local-minimum and per-frame cost problems that the file notes for CPF and ContactOpt. Edges change state by hysteresis: an edge must be supported for a minimum duration before a grasp is declared or released. This keeps grasp state stable through brief occlusions.

There is no egocentric event dataset with moving cameras and mesh ground truth for both hands and objects. We would therefore build one by simulation from HOI4D. HOI4D provides egocentric RGB-D sequences with camera trajectories, object CAD models, and hand and object tracks. We convert its frames to events with the v2e simulator, the same route EvRGBHand and Ev2Hands used with InterHand2.6M. For real-world testing, we would capture a modest set with a head-mounted event camera and motion-capture ground truth, following the EE3D-R protocol from EventEgo3D++. We would deliberately include holds where the hand is still relative to the head while the head turns. The method requires object templates, which is the CAD dependence that the file flags as a limitation of CPF. We accept that dependence here and treat template-free objects as out of scope.

Baselines

The naive reference is Ev2Hands extended with an object branch and trained on the simulated egocentric data with no ego-motion handling. Its own authors predict this will collapse. Running it measures how much the storm actually costs.

The strongest existing system for egocentric event noise is EventEgo3D++'s learned body/background segmentation with its frame buffer. We port it to hands and objects, train it on the same HOI4D-derived events, and follow it with CPF's contact optimizer. This is the "learn to filter, then fix physics afterward" pipeline, and it is the strongest competitor because both stages are proven components. Fairness comes from four things:

  • the same training events;
  • the same input window length;
  • the same A-MANO and object templates;
  • the same number of refinement iterations at test time, so that our joint optimization gets no extra compute.

Every method also gets an equal hyperparameter search budget on a held-out validation split.

A third baseline compensates for ego-motion analytically from the inertial sensor and keeps only the residual events. It is the inertial version of Vasco et al.'s kinematic subtraction.

Three ablations isolate the proposal's ingredients:

  • Edges. The full method with contact edges disabled, so every entity has an independent motion, as in space–time clustering. This isolates the graph.
  • Physics gate. Edges driven only by network probabilities, without the mesh plausibility terms. This isolates physics as the arbiter of structure.
  • Occlusion residual. The method with $o(T_i)$ removed, which tests whether reading holes in the storm matters.

What counts as beating the strongest baseline is lower hand and object pose error together with lower penetration depth and intersection volume. Both must hold on the subset of frames where the foreground moves with the camera; an aggregate win alone does not count.

For calibration we can quote existing numbers. On static-camera Ev2Hands-R, Ev2Hands reports a 3D joint AUC of 0.64 and a fast-motion collision rate of 4.09%. On the synthetic EE3D-S benchmark, EventEgo3D++ reports 98.67 mm MPJPE. For egocentric event-camera hand–object reconstruction under ego-motion, no reported result was found on Lacuna, so the baselines above will be run by us.

Why not just…?

Why not just subtract ego-motion with an inertial sensor and then run Ev2Hands with a contact loss? For free-moving hands this may work well, and the experiment will show it. The inertial baseline is run precisely for that reason. Subtraction removes the background, but it also removes the co-moving hand and cup, because they share the camera's motion. Vasco et al.'s kinematic mapping has the same blind spot. We expect the inertial pipeline to match ours on frames where the hands move independently and to fail on still-hold frames. If it matches us on still-hold frames as well, the graph is unnecessary and we would say so.

Why not just add RGB frames, as EvRGBHand does? RGB is a real fallback for co-moving objects, because a held cup is sharp in RGB precisely when it is still relative to the camera. It brings back blur during fast manipulation and adds a second sensor, though. Its MPJPE gains of 12–18 mm were reported under lighting stress, not head motion. We will run an RGB-fused variant as an upper reference. The question is whether the event-only contact graph closes most of the gap at event rates. If it does not, the honest conclusion is that ego-motion is better handled by sensor fusion than by structure.

Why not just learn the segmentation end-to-end, as EventEgo3D++ does, and skip the explicit graph? This is the strongest baseline above. The evidence against post-hoc or purely learned physics comes from ContactOpt and CPF. ContactOpt's explicit optimization beat a learned refinement network on HO-3D, bringing MPJPE from 57.7 to 48.1 mm. Grasps without CPF's field fell apart in physics simulation. Neither result tests event segmentation, so the edges-off and physics-gate ablations are what settle whether coupling structure into segmentation beats a learned mask followed by contact fixing.

Proposed Experiments

The decisive check is a stratified evaluation on the real head-mounted captures. Every frame is labeled by whether the hand and object move independently of the head or with it. On each stratum we report:

  • hand MPJPE and joint AUC;
  • object pose error;
  • penetration depth and solid intersection volume, the metrics CPF used on F-PHAB;
  • contact-edge F1 against motion-capture-derived contact;
  • grasp stability when the recovered hand and object are dropped into a Bullet gravity simulation, as in CPF.

The claim holds if our method's error on co-moving frames stays close to its error on independent-motion frames while every baseline degrades sharply. The claim is refuted if the inertial-compensation baseline or the edges-off ablation matches us there.

A second check removes support. The method is rerun on static-camera Ev2Hands-R and should keep the reported 0.64 AUC and roughly 4% collision rate, showing the graph does no harm when there is no storm.

A third check sweeps the compliance margin $\delta$ from 0 to 5 mm. It tests the tension between ContactOpt's soft contact and Ev2Hands' rigid intersection loss in a new setting. We expect a rigid margin to cause edges to flicker on and off and floating to increase, the overcorrection the file reports in the generative setting from Layout-your-3D.

Finally, we evaluate on simulated sequences with injected head-shake of increasing speed. This charts where occlusion-boundary evidence stops being informative, because the background becomes too sparse to outline a hole.