Learning Docking Pose from the Dock Itself: Hindsight Labels for Vision-Only Modular Swarms

Modular swarm robots such as PuzzleBots have removed most of their physical tethers. They dock through passive plastic knobs and hooks instead of powered latches, and they run for ten hours on small batteries. They still cannot leave the lab, because every docking maneuver depends on an external motion-capture system (a room of cameras that tracks markers on each robot) to report where each robot sits relative to its neighbor. Putting a camera on each robot is the obvious fix, and it fails for two reasons. The robots look alike from several sides, so a network that regresses pose averages the look-alike answers into a wrong one. And no one can hand-label thousands of close-range images of robots in the field.

This note proposes that the docking event itself supplies the missing labels. When two robots couple, their relative pose is known exactly from the mechanical design, up to the ways the connectors could have mated. Replaying each robot's wheel odometry backward from that moment gives a relative-pose label for every camera frame in the approach. These labels are set-valued: each one is a small group of poses rather than a single pose. That set is exactly the object the symmetry literature has learned to train against, so a symmetry-aware loss can consume it directly. The result is a pose estimator that learns on the robot, needs no human labels, needs no depth sensor, and needs motion capture only to evaluate it.

Core Claim

A robot's own coupling events plus a short odometry window give metrically accurate, symmetry-set-valued pose labels. Training on them with a loss that takes the minimum over the task's symmetry group lets an onboard monocular estimator support passive docking at close to motion-capture success rates. The estimator should also beat photometric self-supervision, which is known to fail on textureless symmetric parts.

The second half of the claim concerns which symmetry to handle. The relevant group is not the full visual symmetry of the robot body. It is the subgroup $G_{task}$ of body symmetries that map docking features onto functionally equivalent docking features. Visual symmetries outside $G_{task}$, such as a 180° turn that swaps the knob face with the hole face, must be represented as a discrete multimodal prediction. They should not be collapsed. Each attempted dock that succeeds or fails then labels which mode was correct.

Relation to Prior Work

Three self-supervised 6D pose methods motivate this work and also mark its failure region. Self6D refines a synthetic-trained estimator on unlabeled RGB-D data using differentiable rendering and a depth-based alignment loss. It raises ADD-S recall on LINEMOD from 40.1% to 58.9%, but it needs a depth sensor at training time and gets trapped at symmetry seams. Pseudo Flow Consistency drops the depth requirement by keeping only pseudo-labels that agree under optical flow across several rendered views. It reaches 92.2% ADD-0.1d on LINEMOD and 65.4% on Occluded-LINEMOD, and its authors name perfectly symmetric objects as its main failure mode. Pose Consistency and Warp-Alignment uses cycle consistency through rendering and photometric warping between image pairs. It degrades on monochromatic surfaces and leaves rotation about symmetry axes unconstrained.

All three derive their training signal from appearance. We derive it from physical contact and the robot's own motion, so textureless 3D-printed parts do not starve the signal.

The loss design comes from the supervised symmetry literature. PoseCNN's ShapeMatch-Loss scores a predicted rotation by how well the transformed mesh points match the ground-truth mesh. GCPose minimizes a matching loss over a predefined set of equivalent poses. It reaches 67.9% average recall on T-LESS, but only with depth input and upstream masks. Pitteri et al. and Mahendran et al. constrain the design: they show that naive regression against arbitrary symmetric labels converges to a meaningless average rotation. We reuse min-over-group losses. We differ in where the label set comes from (hindsight, not annotation) and in which group is quotiented out ($G_{task}$, not the full visual group).

Gilitschenski et al. predict Bingham distributions, which are orientation distributions that can spread along a symmetry axis. We use a simpler discrete mixture over cosets, because PuzzleBot symmetries are finite. We also avoid the Bingham model's unstable normalization tables.

The sparse-anchor idea comes from Guizilini et al.. They show that about 77 metric LiDAR points per frame, fed through a reprojected-distance loss, anchor an entire self-supervised depth model to KITTI Abs Rel 0.072. Coupling events are our analogue of those sparse points: rare, metric, and cheap.

The application comes from PuzzleBots and Reconfigurable Robot Swarms with Passive Coupling. The latter reports 94.4% coupling success and about 1 mm tracking error under Vicon, a commercial motion-capture system. It also notes that the robots cannot sense whether a joint has engaged except through motor current and encoders. We turn that limitation into the labeling signal.

Research Program

Hardware. Each PuzzleBot receives a low-mass, low-power monocular camera facing forward. The first phase streams images to off-board compute over radio. We state plainly that this replaces one tether with another. The second phase ports a small distilled network to an onboard low-power module, and we report its latency and power draw against the ten-hour runtime.

Coupling detection. A coupling detector reads the signature of the existing "wiggle" locking maneuver: a stall in the drive-wheel encoders plus a current spike. We calibrate it against Vicon in the lab so that its false-positive rate is known before any labels are trusted.

Hindsight labels. Let $T_c$ be the known mated relative pose of the two robots at coupling time $c$. Let $O_{i\to c}$ be the relative motion between frame $i$ and $c$, composed from both robots' wheel odometry. The hindsight label for frame $i$ is the set

$$\mathcal{Y}_i = \{\, T_c\, g\, O_{i\to c}^{-1} \;:\; g \in G_{task} \,\},$$

where $g$ ranges over the task-equivalent connector matings. Odometry drift grows with distance traveled, so labels are used only inside a window before contact. Each label is weighted by the inverse of the drift variance predicted by a wheel-slip model fitted in the lab.

Model and loss. The estimator outputs a mixture over the cosets of $G_{vis}/G_{task}$, where $G_{vis}$ is the visual symmetry group of the body. Each mixture component carries a pose estimate. Training minimizes

$$\mathcal{L}_i = w_i \min_{y \in \mathcal{Y}_i} d\big(\hat{T}_i^{(k^*)}, y\big) - \log \pi_{k^*},$$

where $\hat{T}_i^{(k)}$ is the pose predicted by component $k$, $\pi_k$ is its mixture weight, $k^*$ is the component consistent with the label, $d$ is a mesh-point distance in the style of ShapeMatch, and $w_i$ is the drift-based weight.

Using failed docks. A failed dock after committing to mode $k$ is also data. It gives a negative label, which lowers $\pi_k$ for the frames in that approach.

Pre-training and control. Pre-training uses CAD renders of the robot. The distributed model-predictive controller from the original system, which plans each robot's next moves over a short horizon, consumes the top mixture mode. When the mode probabilities are close, it executes an information-gathering sidestep before committing to a dock.

Baselines

Naive reference. The do-nothing reference is the original system's blind wiggle-and-search behavior with no pose feedback beyond odometry after a rough initial placement. It bounds how much vision contributes at all.

Upper reference. The same controller running on Vicon feedback is the target. The reported figure is 94.4% coupling success, from Reconfigurable Robot Swarms with Passive Coupling.

Strongest learned competitor. This is Pseudo Flow Consistency run on the same robot images, starting from the same CAD-pretrained network and given the same number of unlabeled approach videos and GPU hours. Its best reported number is 65.4% ADD-0.1d on Occluded-LINEMOD, from Pseudo Flow Consistency. No reported result was found on Lacuna for it on T-LESS or on robot docking.

Supervised symmetry reference. A CAD-trained estimator with ShapeMatch-Loss serves as the sim-only reference. For context, the rotation-normalization method of Pitteri et al. reports 41.27% VSD recall on T-LESS.

Fiducial baseline. Each robot carries AprilTag markers on its faces, and pose comes from a standard marker detector with a perspective-n-point solver (which recovers pose from known 2D–3D point matches).

Ablations. Four ablations isolate the key ingredients:

  • Hindsight labels with a single arbitrary label per frame instead of the set $\mathcal{Y}_i$, which tests for the averaging collapse.
  • Quotienting by the full $G_{vis}$ instead of $G_{task}$, which should cause wrong-side docks.
  • Removing negative labels from failed docks.
  • Removing drift weighting.

Fairness and what counts as a win. Every method gets the same images, the same backbone, and a matched tuning budget. Beating the strongest competitor means higher closed-loop coupling success and lower Vicon-measured pose error at contact range.

Why not just…?

Why not just put fiducial tags on the robots? This is the real threat. On rigid tabletop robots, tags may simply work. The note from Origami-based integration about harsh environments is weak evidence at PuzzleBot scale. Our concern is geometric. On a roughly 50 mm body viewed by a low camera, the tag shrinks to a few pixels at long range and leaves the field of view or is foreshortened in the final centimeters, which is where docking is decided. We run tags as a full baseline across approach distances. If tags match our coupling success, we report that tags suffice for rigid modules. In that case the hindsight-labeling contribution narrows to untagged or deforming robots.

Why not just train on synthetic renders? Self6D shows that a synthetic-only estimator reaches 40.1% ADD-S on real LINEMOD images. Flexible TPU parts, which are rubbery and non-diffuse, are harder to render than LINEMOD objects. The sim-only ShapeMatch baseline measures this gap directly, and our method uses the same network as its starting point, so the gain from hindsight labels is isolated.

Why not just swap a symmetry-aware loss into existing self-supervised refinement? That ablation is included as "Pseudo Flow plus min-over-group loss." Our expectation is that it still stalls. Its pseudo-labels come from flow and photometric agreement, and on textureless symmetric parts that agreement is either degenerate or confidently wrong, as the authors of Pseudo Flow Consistency and Warp-Alignment report. If it matches our method, then contact-derived labels add nothing, and we would say so.

Why not just use odometry and blind wiggling without vision? Odometry cannot supply the initial relative pose between two robots that have never touched. The naive baseline quantifies how often blind search fails or times out.

Proposed Experiments

T-LESS proxy study. Before building hardware, we validate the label mechanism on T-LESS, which has textureless symmetric parts and recorded camera trajectories. In each sequence, a single frame receives its ground-truth pose as a "contact" label, expressed as a symmetry set. Labels are propagated to other frames using camera poses with synthetic drift injected at several levels. We train an RGB estimator and score it with VSD recall and the per-image pose distributions of BOP-Distrib. We compare against the sim-only ShapeMatch baseline, against Pseudo Flow Consistency, and against the 41.27% VSD recall reported by Pitteri et al. as a supervised reference. The decisive check is a curve of accuracy versus drift level and versus labeled frames per sequence. It tells us how short the odometry window must be before the approach is useful.

Robot study. We then run on four to eight PuzzleBots across flat ground and 5 mm rough terrain. Vicon is recorded for evaluation only. The collection budget is a fixed number of autonomous docking attempts, all of which feed training. We report:

  • closed-loop coupling success against the 94.4% motion-capture reference;
  • pose error at contact range, sliced by distance;
  • the rate of wrong-side docks, which directly measures whether the $G_{task}$ quotient and the negative labels resolve the symmetry;
  • label error, measured as Vicon-checked drift of the hindsight labels.

Success criterion. The idea holds if vision-only docking comes within a few points of motion-capture success, clearly beats Pseudo Flow Consistency and sim-only training under equal budgets, and shows a clear drop in wrong-side docks from the coset mixture. The idea is refuted if the set-valued ablation performs no better than single arbitrary labels, or if tags match it at every approach distance.