Robotizing Human Demonstrations in Event Space

Robots can learn manipulation from videos of people, but first the video has to look like a robot did the work. The current recipe, Masquerade, erases the human arms from each frame, fills the exposed background with a video inpainting model, and draws a rendered robot arm in their place. The weak step is the fill. When an arm moves, it uncovers pixels nobody has seen. Filling these "disoccluded" regions means inventing texture, and that invented texture blurs, tears and flickers exactly during the fast, contact-rich, two-handed moments a robot most needs to learn. We propose doing the same edit on the output of an event camera instead. An event camera is a sensor whose pixels independently report only changes in brightness, with microsecond timing. In that representation the hard part of the edit mostly goes away. Static background behind a moving arm produces no events at all, so the correct content of most of the uncovered region is simply "nothing." The only events that must be synthesized lie along the edges of the inserted robot.

Put in two sentences: inpainting a hole in a camera image is ill-posed because missing texture has to be invented, but inpainting a hole in an event stream is mostly well-posed because a static uncovered background is supposed to be silent. We test whether this lets human two-handed video be converted into robot training data that is cleaner, keeps more of the fast and occluded frames, and gives better bimanual policies than pixel-space editing.

Core Claim

An event camera fires at pixel $x$ at time $t$ when the log brightness has changed by more than a contrast threshold $C$ since that pixel's last event:

$$ \left|\log I(x,t) - \log I(x,t_{\text{last}})\right| \ge C , $$

Here $I$ is scene brightness and the sign of the change is the event's polarity. Consider a static scene in which a hand sweeps away and uncovers background. Real events appear only where a moving edge crosses the pixel: the arm's silhouette, plus any background edge it uncovers in that instant. In the interior of a disoccluded region, the true event stream after the edge passes is empty.

Removing the human and inserting a robot in event space therefore requires three things. First, delete events attributed to the human arm. Second, synthesize events along the rendered robot silhouette, using the equation above with a background brightness estimate. Third, leave everything else untouched. Errors in the background estimate change only the polarity and density of events at robot edges. They never create the texture flicker that frame-by-frame inpainting introduces.

Our claim is that this asymmetry makes the edited data measurably closer to real robot event data and more useful for training two-armed policies. The advantage should be largest on the subsets where pixel inpainting fails: fast motion, hands crossing, and the moment a hand covers a target just before touching it.

Relation to Prior Work

The work that motivates this is Masquerade. It robotizes 675,713 Epic-Kitchens frames using hand pose from HaMeR, arm segmentation with SAM2, and the E2FGVI inpainter, then co-trains a diffusion policy with 50 robot demonstrations per task. It reports a 74% average success rate on three long-horizon bimanual tasks in unseen scenes. Its own limitations point at the editing step: blurred and distorted backgrounds near contact, and frames dropped when hand tracking fails. We keep its pipeline structure (pose, segment, remove, overlay, co-train) and change only the signal being edited.

The method contribution comes from EA3D. It shows that events, sliced by event count rather than by fixed time, give a blur-free edge scaffold that removes floating artifacts and tearing in novel-view synthesis. We reuse its count-based slicing. We differ in two ways. EA3D uses events to guide RGB generation in static scenes; we edit the event stream itself in scenes containing moving articulated arms.

Two lines of work constrain the design. DECO answers self-occlusion by switching to touch: its tactile adapter raises DECO-50 success from 72.25% to 82.50%. Touch, however, carries no information before contact. Active Vision Might Be All You Need adds a third camera arm. That arm helps on occluded tasks (Thread Needle rises from 30% to 52%), but the larger action space hurt simpler tasks. Both are hardware answers to occlusion; ours is a data answer, and it is compatible with either.

Embodiment-invariant approaches avoid editing pixels altogether. H-RDT pretrains on 829 hours of EgoDex using a 48-dimensional wrist-and-fingertip action vector, and Flow as the Cross-Domain Manipulation Interface tracks object motion independently of the hand. These are the main conceptual competitor, and we treat them as a baseline rather than as background.

On the event side, Event-Based Robotic Grasping Detection shows that binned event encodings support 20 ms single-arm grasp detection. Its observation that smooth, low-contrast objects emit few events is a known failure mode we must measure.

Research Program

Data collection. Building the pipeline needs data that does not yet exist at scale: human two-handed demonstrations recorded with an egocentric DAVIS-style sensor, which outputs aligned intensity frames and events from the same pixels. We will collect a modest corpus in kitchen-like scenes, targeting tens of hours. It will cover Masquerade's three tasks plus a set chosen to stress occlusion and speed: hands crossing over a target, one hand holding while the other moves fast, and reach-to-grasp where the hand covers the object before contact.

Editing pipeline. Hand poses come from HaMeR on the intensity frames. Arm masks come from SAM2, propagated at event rate using the event stream. Events inside the swept arm mask are deleted. The robot is rendered following the retargeted poses, using Masquerade's kinematic retargeting so this variable stays fixed. Robot-edge events are synthesized from the threshold equation above, using the rendered robot's brightness against a background estimate. That estimate is taken from intensity frames in which the region was visible, earlier or later in the clip. The background is used only at robot edges, so a crude estimate should suffice; we test this directly by ablation.

Fast-motion frames. When HaMeR fails on a blurred frame, we interpolate the pose across the gap and keep the frame only if the arm silhouette implied by that pose agrees with the observed events. This rule decides which fast frames survive rather than discarding them all, and we report how many frames it recovers relative to Masquerade's frame filtering.

Policy. The policy reads count-sliced event voxel grids, which are stacks of event counts over a few sub-windows so that motion direction is preserved. Event cameras go blank when nothing moves, so the input also carries a decaying per-pixel record of recent activity and a low-rate intensity frame. A second variant drops the intensity frame, so we can tell what events alone contribute. The encoder is initialized from a joint RGB-and-event pretrained model (CM3AE). It is pretrained on robot-keypoint prediction over the edited data, as Masquerade does, then co-trained with a diffusion policy on 50 robot demonstrations per task. The robot is a two-arm platform carrying the same sensor in a matched head-mounted position.

Baselines

The naive reference is a policy trained only on the 50 robot demonstrations per task, with no human data. We run it with both the event-input and RGB-input versions, so we can separate whether events help on their own from whether edited human events help. Masquerade's scaling study, where Stack Pots rises from 2% to 68% as edited human data increases from 0% to 100%, shows how low this floor can be in unseen scenes.

The strongest existing method is Masquerade's RGB-space editing. Its reported number is a 74% average success rate on Stack Pots, Scrape Potato and Sweep Chilis across unseen scenes (Masquerade). That figure is on their hardware, so we re-run their pipeline on our robot. We apply it to the intensity frames of our own human corpus, so both methods see identical demonstrations, retargeting, robot demonstration counts, encoder size, pretraining steps and hyperparameter search budget. Beating it means matching it within noise on its three tasks and clearly exceeding it on the occlusion and speed subset. We consider ten points or more on that subset meaningful, with at least 20 trials per task per scene.

The ablation that isolates the key ingredient is "edit in RGB, then simulate events." We run Masquerade's RGB edit and convert the edited video to events with the same threshold model. Its policy input is identical to ours, but the hole was filled in pixel space. If this matches our method, the benefit comes from the event representation and not from editing in event space, and the core claim fails.

A second ablation feeds raw, unedited human events, with the hands still present. This tests whether the human-versus-robot appearance gap is already small once color and texture are discarded.

The embodiment-invariant competitor is H-RDT-style pretraining on hand keypoint actions from the same human corpus, followed by the same robot fine-tuning. It sees the same human motion without any pixel editing. For reference only, H-RDT reports 52% on real-world towel folding against 0% without human data (H-RDT).

Why not just…?

Why not just swap E2FGVI for a stronger video diffusion inpainter? This is the most serious alternative, because generative inpainters keep improving. The evidence in the file points the other way on geometry: RGB-only generative models warp structure and produce floaters when the views they are given are sparse, which is the motivation for EA3D and FreeFix. Still, a better inpainter could close much of the gap. We will therefore include the strongest available diffusion inpainter as a third editing condition inside the "edit in RGB, then simulate events" ablation. Its frame-to-frame flicker shows up directly as spurious events inside disoccluded regions, which our offline metric measures. If its flicker rate and policy success match ours, we will report that event-space editing is unnecessary.

Why not just record faster RGB, or add more cameras? A high-frame-rate camera reduces blur, but its disoccluded regions still need invented texture, so it does not touch the core problem. More views multiply image tokens. TwinVLA needed mixture-of-experts routing, which saved 21% of memory, and attention re-weighting just to handle extra camera streams. A third camera arm helped occluded tasks but hurt simple ones (Active Vision Might Be All You Need). We include a robot-only high-frame-rate RGB policy, so that sensor speed alone is not mistaken for the effect of the editing.

Why not just simulate events from existing large video datasets and skip new collection? Events simulated from 30 Hz video contain no timing information the frames lacked, so they cannot recover blurred motion. Simulation does test the editing logic at scale, though, so we run it on EgoDex-scale video as a secondary experiment. If simulated-event editing at large scale beats real-event editing at our small scale, that would be a strong argument that the real event camera matters less than the volume of data, and we would say so.

Why not just skip pixels and pretrain on hand keypoints? Keypoint pretraining is cheap and avoids editing entirely, and H-RDT shows it transfers. It cannot, however, teach the visual encoder what a robot arm covering a target looks like, which is precisely the information needed in the pre-contact blind phase. The keypoint baseline above settles this empirically. The two approaches can also be combined, and if they are complementary we will report the combination.

Decisive Checks

The first check is offline and cheap. On held-out static-scene clips, we measure the ghost-event rate: events per pixel per second inside disoccluded regions more than a few pixels from any robot edge. The correct value is near the sensor noise floor. We also measure a distance between the encoder embeddings of edited human event streams and of real robot event streams recorded in the same scenes. Event-space editing should achieve a lower ghost rate and a smaller distance than "edit in RGB, then simulate." If it does not, the policy experiments are not worth running.

The second check is frame retention. We count the fraction of fast-motion and hands-crossing frames that survive editing under each pipeline.

The third check is policy success on robot trials. We report the Masquerade tasks separately from the occlusion and speed subset, and within that subset we also report success on the approach phase alone: whether the gripper reaches a pre-grasp pose within a set tolerance while the target is hidden. That is where the event pipeline should win if the claim holds.

We commit in advance to three readings. Raw unedited events matching edited events means the embodiment gap in event space is small and editing is unnecessary. RGB-edit-then-simulate matching our method means the representation helps but the editing location does not matter. Low-contrast objects failing in both event conditions marks the limit of the approach and argues for keeping the intensity channel.