Coded Motor Dither for Annotation-Free Robot Self-Segmentation

A robot that has to learn to see its own body usually gets help it should not need: a human who draws masks around the arm, a factory CAD model projected into the image, or an empty, static workspace in which anything that moves must be the robot. This note proposes a way to remove all three. The robot adds a small, private, pseudo-random wiggle to its joint commands, a "dither code" that no one else in the scene knows. It then labels as "self" every pixel whose motion tracks that code over a few seconds. Other people, other robots, and moving objects do not know the code, so their motion averages out, however much they happen to move in step with the arm. The resulting masks, with each pixel also tagged by which joints move it, become free training labels for a segmenter and for policies that need to know where the body is.

The idea borrows from lock-in detection in signal processing and applies it to the oldest idea in robot self-recognition, visuomotor contingency: the self is whatever moves reliably when you command it. Earlier contingency methods established the principle on sparse features and clusters but never produced dense masks in a busy scene. Modern annotation-free robot segmenters produce dense masks but need a static background or a CAD model. Coding the robot's own motor output combines the two: dense, per-pixel self-labels obtained in the presence of moving distractors, from sensors every arm already has.

Core Claim

A known, zero-mean, pseudo-random perturbation injected on each joint lets a robot attribute image motion to itself with confidence that grows with observation time, independent of what else moves in the scene. Concretely, let $c_j(t)$ be the dither code on joint $j$ (small amplitude, broadband, with codes for different joints orthogonal), and let $m_p(t)$ be the optical-flow motion at pixel $p$, band-pass filtered to the dither frequencies. The ownership score of pixel $p$ for joint $j$ over a window of $T$ frames is

$$ s_{pj} = \frac{1}{T}\sum_{t=1}^{T} \frac{\langle m_p(t),\, \hat{d}_{pj}(t)\rangle\, c_j(t)}{\sigma_m \sigma_c}, $$

where $\hat{d}_{pj}(t)$ is the unit image direction in which pixel $p$ moved during recent high-code frames of joint $j$ (estimated online), and $\sigma_m, \sigma_c$ normalize the two signals. For any motion source independent of $c_j$, such as a person, a second identical arm, or a rolling object, $s_{pj}$ is an average of zero-mean products, so its spread shrinks as roughly $1/\sqrt{T}$ by the central limit theorem. Pixels on the robot keep a nonzero mean. The claim is testable: the self/other separation margin should grow with the square root of the dither window, and the method should hold up against distractors that mimic the arm's task-level trajectory. Plain correlation with joint velocity fails in exactly that mimicry case. A second output comes almost for free. A pixel on link $k$ responds to the codes of every joint upstream of it, so the set of codes a region responds to reveals the kinematic chain from vision alone. This is the visual analogue of the joint–sensor dependency matrix that Robot Body Schema Learning from Full-body Extero/Proprioception Sensors builds from body-mounted inertial sensors.

Relation to Prior Work

Three lines of work motivate the proposal. The contingency literature shows the principle and its limits. Robot Self-Recognition Using Conditional Probability-Based Contingency scores pixel clusters by the sufficiency and necessity of action for visual change. Distinguishing Self, Other, and Autonomy From Visual Feedback argues that simple correlation cannot separate the self from a concurrently moving autonomous agent, and uses how accelerations propagate through the scene instead. Neither produces dense masks or trains a network. We keep their logic and make the action side controllable: when the robot designs its own excitation, statistical independence from other agents is guaranteed, rather than inferred from acceleration patterns.

The annotation-free segmenters supply the dense-label goal and show the bootstrapping problem. Self-Supervised Object-in-Gripper Segmentation from Robotic Motions trains an arm segmenter from optical flow while the arm moves in an empty, static workspace. Self-Supervised Surgical Tool Segmentation using Kinematic Information projects a CAD model and refines it with iterative boundary optimization. Moving Object Segmentation Using Motor Signals uses motor commands to predict background motion under a moving camera. We drop the static-scene assumption of the first, the CAD model of the second, and the sparse-feature, camera-motion framing of the third.

Two consumers of self-masks set the downstream stakes. Efficient RL via Disentangled Environment and Agent Representations (SEAR) speeds up visual reinforcement learning by predicting a robot mask from an agent-only latent, but takes the masks from a simulator or an external segmenter. It also reports tolerance to coarse masks, which makes our noisier labels plausible as a drop-in source. Raising Body Ownership in End-to-End Visuomotor Policy Learning via Robot-Centric Pooling (RcP) gates visual features with joint angles and survives identical-arm distractors. Its alignment is purely spatial, on a 7×7 feature map, and its authors note that it neither uses temporal contingency nor resolves heavy 2D overlap. Our score is temporal and per-pixel, so it targets exactly that failure.

Research Program

The system has three parts. The first is the excitation layer. A zero-mean pseudo-random code, on the order of one to two degrees at a few hertz, is added to each joint on top of whatever task or babbling motion is running, with codes orthogonal across joints. We will measure how much the dither degrades task tracking, and choose amplitude as the smallest value that reaches a target separation margin within a fixed window.

The second is the labeler. It runs an off-the-shelf flow estimator, band-pass filters the flow, computes $s_{pj}$ over sliding windows, and thresholds the result into per-pixel self and per-link masks. To get clean boundaries we let the contingency score choose among grouping proposals rather than supply the grouping. Class-agnostic region proposals from a promptable segmenter, or superpixels as a fallback, are each assigned the mean ownership score of their pixels. This matters because the flow-based score is noisy at the edges, and it turns the strongest skeptic's baseline (see below) into a component rather than a competitor.

The third is distillation. A segmenter conditioned on the current joint angles is trained on the harvested labels, so that at deployment it runs without dither and without flow. It is retrained whenever the self-model's predictions and the dithered contingency disagree, for example after a tool swap. The analysis part of the program tests the $1/\sqrt{T}$ margin prediction, measures how kinematic-chain recovery from code responses degrades with occlusion, and checks whether dither on a single joint suffices to label everything downstream of it.

The known limitation is stated up front. When the arm is out of view, as with a wrist-mounted camera, there is nothing to label. We do not claim to address that egocentric blind spot.

Baselines

The naive reference is RGB change detection during motion. Self-Supervised Object-in-Gripper Segmentation from Robotic Motions reports it at 31.18% mean intersection-over-union (mIoU) on its YCB-Video in-gripper setting. We will rerun it on our scenes with the same frames. It serves as the floor, and it shows how much of the problem plain motion already solves.

The strongest existing annotation-free method for the arm itself is the flow-during-motion bootstrapping of that same paper. Its full pipeline reaches 75.80% mIoU on the object-in-gripper task, and it is the closest thing to "discover your own mask without models." We will run it exactly as published: its flow network, its segmentation backbone, and its background-pasting augmentation. We will use the same robot, the same number of motion frames, and the same backbone and training budget as our distilled segmenter, in two conditions: its intended static workspace, and our dynamic scenes. No reported number was found on Lacuna for arm self-segmentation mIoU in dynamic scenes, so that condition is new measurement. Beating it means matching it within noise in the static condition and exceeding it by a clear margin, which we pre-register as at least 15 mIoU points, once people or a second arm move in the background.

The model-based competitor is CAD/URDF projection, with and without learned refinement, following Self-Supervised Surgical Tool Segmentation using Kinematic Information. That paper reports 0.56 intersection-over-union (IoU) for raw kinematic projection versus 0.86 after training, against 0.87 for full supervision on its Phantom 1 data. We give it the robot's URDF and a standard hand-eye calibration, and we score it on the same held-out masks. We do not expect to beat it when the model is accurate. The target is parity with it on an unmodified robot, and a clear win after an unmodeled tool or extension is attached.

For ownership rather than segmentation, the competitor is the RcP policy. It reports 92% success in the real self-distractor scenario, against below 15% for spatial-softmax and R3M policies. We run it on our reimplementation of its CoppeliaSim reaching setup with the identical second UR5 arm, adding a harder variant in which the distractor arm replays the ego arm's trajectory with a short delay and overlaps it in the image.

The key ablation removes the code. It keeps the same pipeline but correlates flow with the robot's ordinary joint velocities, without injected dither. This is essentially dense Godby-style contingency, and it isolates whether the private code, rather than the contingency idea itself, is what buys robustness.

Why not just…?

Why not just project the URDF into the image? When the model and calibration are right, this is hard to beat. We say so and test it head-on on an unmodified robot. The evidence against relying on it comes from the cases the method cannot see. Raw projection reached only 0.56 IoU in Self-Supervised Surgical Tool Segmentation using Kinematic Information because of calibration and kinematic error. Full-Body Visual Self-Modeling of Robot Morphologies shows fixed models breaking when a motor jams or the body changes. Our tool-swap and attached-extension conditions settle whether coding adds anything beyond a good model. If it does not, the contribution narrows to robots without trustworthy models.

Why not just prompt a foundation segmenter to find "the robot"? A promptable segmenter draws good boundaries, but it has no notion of which arm is this robot. The R3M-based policy's collapse below 15% with an identical distractor arm in Raising Body Ownership in End-to-End Visuomotor Policy Learning via Robot-Centric Pooling is the relevant warning about general-purpose features. This alternative is a real threat for grouping, so we adopt it: its proposals are the regions we score. The experiment that decides its value is the identical-arm condition. There we compare its text- or point-prompted masks against the same proposals selected by contingency score.

Why not just empty the workspace and bootstrap from flow, as in the in-gripper work? In a lab this works, and that is why it is our main baseline. Deployment settings, such as shared workcells, homes, and dual-arm stations, rarely allow a clean, static scene, and every tool change would need one again. The dynamic-scene comparison measures the cost of that assumption directly.

Why not just correlate flow with joint velocity, without a special code? This is the ablation. The prediction is that it performs about as well as coding when distractors move independently of the arm, and fails when a distractor imitates the arm, as in the delayed-replay arm. This is the failure mode Distinguishing Self, Other, and Autonomy From Visual Feedback identifies for correlation. If the uncoded version matches coding even under mimicry, the dither is unnecessary and the proposal reduces to dense contingency. We would report it as that.

Proposed Experiments

Segmentation quality is measured in two settings. The first is a real arm with hand-labeled test masks (labels used only for evaluation), recorded in four conditions: a static background, people walking behind the arm, a second arm on its own schedule, and a second arm replaying the ego arm's trajectory with a delay. The second is the in-gripper YCB-Video protocol, where arm masks are an intermediate step whose quality should carry through to object masks. Metrics are mIoU for the self mask, link-level IoU for per-joint masks, and the separation margin between self and distractor pixels as a function of window length $T$, fitted against the predicted $1/\sqrt{T}$ curve. The first decisive check is the delayed-replay condition. Coded contingency should keep its margin while uncoded correlation and flow-in-empty-workspace bootstrapping degrade. If it does not, the core claim is wrong.

Downstream value is measured twice. In simulation, SEAR is trained on Meta-World tasks with the simulator masks swapped for our harvested masks, and its learning curves are compared against simulator-mask SEAR and plain DrQ-v2. The second decisive check is that harvested masks recover most of the gain from ground-truth masks. For reference, Meta-World reports 85% multi-task success for multi-headed SAC on the ten-task suite. In the CoppeliaSim identical-arm reaching setup, a policy whose visual features are gated by our distilled self-segmenter is compared against RcP, with RcP's reported 92% as the reference and the delayed-overlap variant as the stress test.

The kinematic-structure analysis checks whether the pattern of codes each region responds to recovers the correct parent–child ordering of links, both on the real arm and on at least one reconfigured variant, with and without partial occlusion of a link. Finally, we report the dither's cost: task tracking error and cycle-time overhead at the chosen amplitude. A labeler that disturbs the task it is meant to support would not be deployable.