Contrastive pretraining teaches an image encoder by pulling two views of the same moment together and pushing every other image in the batch away. That rule quietly assumes every other image is a different thing. Cameras mounted on a robot's wrist break the assumption badly. Because the camera rides with the gripper, thousands of frames from unrelated episodes look almost identical: the same empty table sliding past, the same bin rim, the same closed fingers holding a block. Some of these look-alikes really are the same situation and should not be pushed apart. We call these "false negatives." Others only look alike. Moving down to grasp and moving down to place can produce nearly the same wrist image while requiring different next moves, and these are precisely the hard cases the encoder must learn to separate. Existing false-negative filters decide which pairs to spare using the encoder's own visual similarity. On wrist streams that criterion cannot tell the two cases apart.
This note proposes a different criterion. Eye-in-hand viewing combined with actions expressed relative to the gripper puts every demonstration in the same local coordinate frame, so the demonstrator's next few gripper-frame motions (the action chunk) can be compared directly across episodes, positions, and scenes. We use closeness of these future relative motions, including the open/close command, to decide which in-batch negatives to drop from the contrastive loss. The encoder never sees this signal as a regression target. It only controls which pairs are allowed to repel. Because the criterion does not depend on the encoder, it avoids the confirmation-bias trap that forces clustering-based methods to ramp detection up slowly. It also needs no global clustering pass and no calibrated goal positions.
Core Claim
In wrist-camera streams, visual similarity is a poor proxy for functional equivalence, while similarity of the future gripper-frame motion is a good one. Removing negatives flagged by the motion proxy from the contrastive denominator should produce wrist encoders that train better low-data manipulation policies than three alternatives: unfiltered time-contrastive pretraining, and the visual-similarity and clustering filters imported from image classification. The effect should be larger for wrist views than for third-person views, because aliasing is what makes wrist frames dense in false negatives.
Where This Sits
The false-negative literature supplies both the method and a constraint. Incremental False Negative Detection for Contrastive Learning (IFND) finds false negatives with multi-scale k-means on encoder features and removes them from the denominator of the loss. It shows that this removal ("elimination") is more stable than treating suspects as extra positives ("attraction"). It also needs a detection rate ramped from 0 to 100% because early features are unreliable, and periodic whole-dataset clustering that cost about 3.5 of 60 training hours. We keep elimination but swap the detector for one that is reliable from the first step. Our pairing rule borrows the mutual-nearest-neighbor idea used by Seong et al. for segmentation, so we can avoid a hand-set threshold. Similarity-threshold filters such as that of Tan et al. are the simplest competitor, and we test them directly.
The robot-representation literature motivates the gap. Time-Contrastive Networks and R3M repel frames that are far apart in time, which penalizes the revisited states common in pick-and-place. Neither filters these pairs, and both use third-person or head-mounted video. Polybot is the closest system. It aligns wrist-camera features across robots with a triplet loss, but it defines positives from calibrated gripper-to-goal position and takes negatives without filtering. Our proposal replaces the privileged goal geometry with the robot's own relative motion and uses it only to veto negatives. VIP imposes a monotone value ordering that multi-stage tasks violate, and we do not inherit it.
The eye-in-hand literature supplies the reason the proxy should work. Vision-Based Manipulators Need to Also See from Their Hands shows that wrist views remove global distractors (92–100% lower out-of-distribution failure in simulation) but suffer partial observability, which is the source of the aliasing we exploit. A Practical Guide for Incorporating Symmetry in Diffusion Policy shows that wrist views plus relative actions give an approximately SE(3)-invariant problem, meaning one that is unchanged when the whole scene is rigidly moved or rotated. That same invariance makes gripper-frame action chunks comparable across episodes. PALM makes a related point with synthetic hand-centric crops and camera-relative proprioception, but it trains with supervised behavior cloning only, so it has no contrastive objective and nothing to filter.
Research Program
We pretrain a ResNet encoder on wrist frames with an InfoNCE objective, the standard contrastive loss. The positive for each anchor is an augmented copy of the same frame plus a frame a few steps later in the same episode. For each frame $i$ we form a descriptor $m_i$ that stacks the next $H$ relative end-effector displacements and rotations, expressed in the gripper frame, together with the gripper command. Any robot dataset with proprioception can produce this descriptor without calibration.
A negative $j$ is removed from anchor $i$'s denominator when $i$ and $j$ are mutual $k$-nearest neighbors in $m$-space within the current batch-plus-queue. The loss is then
$$ \mathcal{L}_i = -\log \frac{e^{s(z_i, z_i^+)/\tau}}{e^{s(z_i, z_i^+)/\tau} + \sum_{j \notin \mathcal{F}_i} e^{s(z_i, z_j)/\tau}}, $$
where $z$ are encoder embeddings, $s$ is cosine similarity, $\tau$ is a temperature, and $\mathcal{F}_i$ is the set of negatives flagged by the motion proxy. The detector is fixed by the data, so the full rule applies from the first step with no acceptance ramp and no offline pass.
Visually similar frames with different future motions stay in the denominator. We additionally test whether upweighting them helps. A variant gates elimination on both motion closeness and visual closeness, so we can see whether visually different but motion-matched pairs, such as approaching two different objects the same way, should be spared or kept.
Before any policy training, we measure detection quality directly. MimicGen and robosuite give exact object poses relative to the gripper and a subtask segmentation for every frame. From these we label each in-batch pair as functionally equivalent or not, and report precision and recall for four detectors: feature-similarity thresholding, IFND-style clustering, our motion proxy, and random. We run this for wrist and third-person streams alike. The labels come from the simulator only for evaluation and are never used in training.
Baselines
The naive reference is a diffusion policy with relative actions on wrist plus third-person images, using an ImageNet-pretrained ResNet trained end to end with standard augmentation. This is the practical setup of Wang et al., who report that relative actions alone give a 5–7% gain and equivariant encoders an average 9.1% gain across 12 MimicGen tasks. The file gives those gains but no absolute success rates. We will reproduce the setup and additionally report the equivariant-encoder variant as an upper reference. Beating this baseline means a statistically reliable gain in average success at 100 or fewer source demonstrations per task.
The second reference is frozen R3M features, which the paper reports beat CLIP and MoCo by over 10% on simulated suites. It tests whether in-domain wrist pretraining is needed at all.
The strongest direct competitor is IFND-style elimination: the same encoder, data, loss, and epochs, but with k-means detection and IFND's ramp, given the same hyperparameter search budget as our method over cluster counts and the ramp schedule. Its only reported number is 69.7% ImageNet linear top-1. We found no reported result on Lacuna for IFND or any false-negative filter on wrist-camera robot data, so we establish its numbers ourselves.
Alongside it we run unfiltered time-contrastive pretraining in the style of TCN and R3M, and a tuned cosine-threshold filter.
Four ablations isolate the key ingredient. A random-elimination control drops the same fraction of negatives, which separates "fewer negatives" from "the right negatives." An attraction variant turns proxy matches into positives, testing whether IFND's elimination-over-attraction finding transfers to robot data. The visual-gate variant described above tests the joint criterion. Finally, a third-person-only version of every method tests whether the gain is specific to eye-in-hand views.
All methods share the same pretraining frames, compute, and downstream policy recipe, and results are averaged over three seeds.
Why not just…?
Why not just train behavior cloning end to end with augmentation? On wrist views with relative actions this is a strong and honest baseline. Wang et al. show it already captures much of the symmetry benefit. Against that, PALM reports vanilla behavior cloning degrading 71% in simulation and 77% on a real robot under workspace and viewpoint shifts, and R3M reports that learning from scratch lags pretraining by more than 20%. Pretraining matters most when demonstrations are scarce, so we sweep the number of demonstrations. If end-to-end training matches us at every scale, the proposal fails.
Why not just predict the actions as a pretraining target, since the proxy already uses them? This is the most serious threat. Regressing action chunks gives the encoder the same information more directly. Our case rests on two things: regression averages over the multiple valid ways demonstrators solve a task, while a mask does not, and a mask cannot overwrite visual structure the way a regression head can. Neither is established. We therefore run an action-chunk-prediction pretraining baseline, and a combined version, under an identical budget. If prediction alone matches us, we will report that motion supervision, not false-negative filtering, is what matters.
Why not just filter by feature similarity or IFND clustering, which need no actions? These methods decide from appearance, and wrist aliasing is exactly where appearance misleads. The pair-labeling study tests this directly: if visual detectors reach precision comparable to the motion proxy on wrist streams, our premise is wrong and the cheaper method should be preferred.
Why not just drop negatives altogether with a negative-free objective? That removes false negatives but also the aliased hard negatives we argue are valuable. Adding a BYOL-style variant, which learns without any negatives, under the same budget would settle it. We treat it as an optional extra baseline if compute permits.
Proposed Experiments
Pretraining uses wrist and agent-view frames from MimicGen-generated robosuite data, plus the multi-robot wrist-camera data of Polybot, whose reported 19% gain from contrastive alignment gives a real-data point to improve on. We will add BridgeData V2 trajectories if suitable views are available. Downstream evaluation is diffusion-policy success on the 12 MimicGen tasks used by Wang et al., at 25, 50, and 100 source demonstrations. We also run Meta-World out-of-distribution evaluations following Hsu et al., reporting failure rates under changed table height, distractors, and textures.
Three decisive checks follow from the claim. First, on wrist streams the motion proxy must have clearly higher precision than the visual detectors, and that gap must shrink on third-person streams. Second, our method must beat both random elimination at a matched rate and action-prediction pretraining; beating only the unfiltered baseline would not support the claim. Third, the gain over unfiltered pretraining should grow on multi-stage tasks with revisited states, such as pick-place and assembly, and should be smallest on single-phase lifting. If the wrist-versus-third-person gap or the multi-stage trend fails to appear, the false-negative explanation does not hold, whatever the headline success numbers show.
References
- Incremental False Negative Detection for Contrastive Learning
- Time-Contrastive Networks: Self-Supervised Learning from Video
- R3M: A Universal Visual Representation for Robot Manipulation
- Polybot: Training One Policy Across Robots While Embracing Variability
- VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training
- Vision-Based Manipulators Need to Also See from Their Hands
- A Practical Guide for Incorporating Symmetry in Diffusion Policy
- PALM: Enhanced Generalizability for Local Visuomotor Policies via Perception Alignment
- BridgeData V2: A Dataset for Robot Learning at Scale