A robot policy that maps camera images directly to motor commands has to discover, from a few dozen demonstrations, which pixels matter: the handle it grasps, the peg it inserts, the spot where the object goes. When data is scarce it often latches onto background texture instead. We propose to supervise that discovery with labels the demonstrations already contain. Each time the gripper closes on an object or releases it, the robot's own joint readings, projected through a calibrated camera, mark a single image point that must be task-relevant. That is a free "click", the same kind of sparse label that weakly supervised segmentation research has learned to turn into full object masks. We grow these self-generated clicks into dense masks of the regions the robot interacts with, using optimal transport over the policy's own visual features. The masks then train an auxiliary head on the policy's image encoder. At test time the policy still sees the whole image and needs no clicks, trackers, or segmenters.
Our bet is that this closes much of the gap between two families of methods. Keypoint policies are data-efficient but blind to anything nobody annotated. Pixel policies can see everything but need many demonstrations to learn where to look. The annotation cost of our approach is zero, because the labels come from the gripper. This means any gain in low-data success and robustness to distractors comes purely from better use of data that is already collected.
Core Claim
Gripper open and close events give sparse spatial seeds for free: one point on the grasped object, and one point at the place location. Because a fixed camera sees the object sit still until it is contacted, each seed also labels every earlier frame at no cost. We claim that densifying these seeds into masks and using the masks as an auxiliary target for the visual encoder of an end-to-end diffusion policy will improve success in the low-demonstration regime (10 to 50 demos) and under unseen distractors. We further claim that it does so without the failure mode of keypoint-only policies, which cannot perceive objects nobody annotated. We also claim the densification step itself matters: supervising only the seed pixel should help less. This is the claim most at risk, and the experiments are built to test it.
Relation to Prior Work
The closest work uses sparse spatial labels as the policy's input. Point Policy has a human annotate object keypoints on one frame. It propagates them with frozen correspondence and tracking models and feeds only the 3D point tracks to a transformer, reaching 88% average success on eight real Franka tasks. Because it discards all other visual context, it cannot react to unannotated obstacles, and it breaks when the tracker loses a point. K-VIL similarly extracts keypoints with frozen pretrained perception and hands them to classical motion primitives. We differ on three counts. Our sparse labels are a training signal rather than an input. The labels come from the robot rather than a human. And the visual representation is trained jointly with the action loss rather than frozen. These papers motivate the proposal: they show that knowing which points matter yields large data-efficiency gains. They also constrain it: they show that routing perception through fixed points is fragile.
A second group uses dense intermediate affordances. PALM co-generates masks, contact heatmaps, and progress alongside actions and reaches 91.8% on LIBERO-LONG. To do so it depends on 942 trajectories with exhaustive human affordance annotation. AnchorDP3 obtains its masks from simulator rendering and supervises sparse keyposes. RT-Affordance has a vision-language model predict gripper keypoints once per episode as a visual prompt. We keep the idea that affordance supervision helps, but replace hand-labeled or simulator-given masks with masks grown from gripper events. We also borrow AnchorDP3's observation that grasp and release moments carry most of the decision content, and use those moments as our seed times.
The densification method comes from the sparse-annotation literature. Point2Mask casts pseudo-mask generation from one point per object as an optimal transport problem, meaning a principled way to divide unlabeled pixels among a few labeled anchors, and reports roughly 90% annotation savings. Point-supervised segmentation via objectness regularization and graph-constrained contrastive regularization show that feature-space compactness and shape priors keep propagated labels from leaking into the background. These works also name the risks we must handle: boundary bleed, sensitivity to where the seed lands, and confirmation bias in self-training. Here the seed placement is decided by physics rather than by an annotator, which is a new twist: gripper contact points sit near object edges by construction.
The policy backbone is Diffusion Policy, which generates action sequences by iteratively denoising random noise.
Research Program
For each demonstration, we detect gripper-close and gripper-open events from proprioception. We project the fingertip midpoint at each event into the fixed camera image to obtain a seed $s_k$. We attach the grasp seed to all frames before the grasp and the place seed to all frames. We also add a background seed from pixels far from the arm and both seeds.
On each training frame, we compute a pseudo-mask by entropic optimal transport. Pixels $i$, with features $f_i$ from an exponential-moving-average copy of the policy encoder, are transported to seeds $k$, whose features are $g_k$. The cost is
$$C_{ik} = 1 - \cos(f_i, g_k) + \lambda \,\lVert x_i - x_{s_k} \rVert / D,$$
where $x_i$ is the pixel position and $D$ is the image diagonal. The seed marginals are estimated object areas, updated from the previous round's masks as in Point2Mask. The policy's encoder gains a light mask head trained with cross-entropy on confident pseudo-mask pixels only. The total loss is
$$L = L_{\text{diff}} + \beta\, L_{\text{mask}},$$
where $L_{\text{diff}}$ is the standard denoising loss and $\beta$ ramps up after the encoder has warmed up. Using the slow-moving teacher copy for the transport step, plus the confidence threshold, is our guard against confirmation bias.
Optionally, the predicted mask is concatenated to the image features the diffusion head sees. We ablate this.
The analysis side measures pseudo-mask quality against simulator ground-truth masks. It then relates mask quality to policy success, which tells us whether better masks actually cause better control.
Baselines
The naive reference point is vanilla image-based Diffusion Policy trained on the same demonstrations with the same encoder, augmentation, and training steps. This isolates whether the extra loss helps at all. The strongest published numbers here are Diffusion Policy's 95% success on Push-T, Diffusion Policy, and BC-RNN's 84% on Square and 71.3% on Transport on RoboMimic human data, What Matters in Learning from Offline Human Demonstrations. Those figures come from full datasets. Our claim concerns the regime of 10 to 50 demonstrations, where we will rerun Diffusion Policy ourselves with an identical tuning budget. Beating it means a statistically reliable gain in success across seeds at matched demonstration counts, while staying within noise of it at full data.
The ablation that isolates densification is a seed-heatmap auxiliary. It uses the same gripper-derived seeds, but the head regresses a Gaussian blob at the seed pixel, with no optimal transport. If this matches our method, then the value lies in hindsight seeding and densification is unnecessary.
The strongest threat is a frozen-segmenter variant. It feeds the same seeds as prompts to an off-the-shelf promptable segmentation model and uses the resulting masks as the auxiliary target, again with no optimal transport. We will give it the same seeds and the same loss weight.
The keypoint-input baseline reimplements the Point Policy approach on our tasks. We track the seeds with a frozen point tracker and feed the point tracks, and only the tracks, to the policy. The comparison uses the same demonstrations. No reported result found on Lacuna for Point Policy on RoboMimic, Push-T, or RoboTwin, so all numbers for this baseline will be our own.
Why not just…?
Why not just collect more demonstrations or add augmentation to a pixel policy? This is the default answer, and at full data it probably wins. The evidence that pixel policies struggle when data is scarce or objects are unseen comes from other settings. Track2Act reports 0% for pixel behavior cloning on unseen object types versus 40% for its track-based method. Point Policy reports a 75-point gap over a pixel baseline. Our demonstration-count sweep answers the question directly: it measures how many extra demos vanilla Diffusion Policy needs to match our success at $N$ demos. That exchange rate is the headline number, since our labels cost nothing.
Why not just feed tracked keypoints, as Point Policy does? Keypoint inputs are data-efficient, but the policy is blind to anything off the tracked points and fails when tracking fails. Our obstacle and distractor tests are designed to expose that difference. We insert unseen objects on the path at test time, and we occlude a seed region. If the keypoint baseline matches us on clean scenes but collapses on these, the case for keeping full-image input is made. If it does not collapse, our advantage is limited to not needing a tracker at test time.
Why not just prompt a frozen segmentation model with the seeds instead of running optimal transport? This is a real threat. A strong promptable segmenter may produce better masks than transport over a small policy encoder, especially on textured objects. We run it as a baseline under identical conditions. If it matches or beats our method, we will report that the contribution is hindsight seeding plus mask-as-auxiliary-target, and that the densifier can be any segmenter. Transport keeps two possible advantages. It adapts to the policy's own features, and it needs no extra large model during training. The mask-quality-versus-success analysis will show whether those advantages matter.
Proposed Experiments
We run on RoboMimic Lift, Can, Square, Transport, and ToolHang with proficient-human data, subsampled to 10, 25, 50, and full demonstrations. Push-T is included as a contact-rich case, where we take seeds from the moments the pusher touches the block. The RoboTwin bimanual tasks test two grippers producing interleaved seeds. For each setting we use five training seeds and fixed evaluation rollouts, reporting success rate.
The robustness split adds unseen distractor objects, background textures, and path obstacles at test time. The obstacle split is the setting where keypoint-only input should fail. We measure pseudo-mask intersection-over-union against simulator masks throughout training to watch for drift caused by confirmation bias.
A small real-arm replication with a fixed calibrated camera on three pick-and-place tasks checks whether projection error in the seed breaks the method.
Three results would be decisive. The first is a clear low-data gain over vanilla Diffusion Policy that shrinks toward zero at full data. The second is that our method beats the seed-heatmap ablation, which shows densification matters. The third is that it holds up on the obstacle split where the keypoint baseline fails. A null result on the second, or a loss to the frozen-segmenter baseline, would narrow the claim to "self-labeled seeds are a useful auxiliary signal," which is still worth knowing.
References
- Point Policy: Unifying Observations and Actions with Key Points for Robot Manipulation
- K-VIL: Keypoints-Based Visual Imitation Learning
- PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulation
- AnchorDP3: 3D Affordance Guided Sparse Diffusion Policy for Robotic Manipulation
- RT-Affordance: Affordances are Versatile Intermediate Representations for Robot Manipulation
- Point2Mask: Point-supervised Panoptic Segmentation via Optimal Transport
- Point-Supervised Segmentation Of Microscopy Images And Volumes Via Objectness Regularization
- Graph-Constrained Contrastive Regularization for Semi-weakly Volumetric Segmentation
- Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
- What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
- Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulation