Silhouette Handoff Contracts: Stopping Error Buildup Between Robot Agents

When several robot arms split an assembly job into stages, each stage starts wherever the previous one actually left the parts, not where the plan says it left them. Small offsets from camera calibration, gear backlash, or a part slipping in the gripper carry forward and grow. The overall success rate then falls roughly like a product of per-stage success rates, which is how a pipeline that looks good in simulation can score zero on real hardware. We propose a check at every handoff between agents. Before agent $j$ takes over from agent $i$, the system renders the silhouettes (the 2D outline masks) that the two mating bodies should cast in a wrist camera. It compares them against the segmented image and corrects the relative pose between the two bodies by gradient descent through a differentiable renderer, meaning a renderer whose output mask can be differentiated with respect to 3D pose. The stage is released only when the remaining mismatch falls below a tolerance derived from the CAD mating relation.

The idea has two parts. First, the check is always pairwise and relative: part-to-part or gripper-to-part, as seen from a camera near the contact. This makes errors in the global camera frame largely cancel, and the cost grows with the number of mating pairs, not with the joint state of every arm and part. Second, the gradient of the mismatch says which agent's body is off. When a check fails, only the responsible agent re-plans, instead of the whole multi-agent dialogue running again.

Core Claim

Treating inter-agent handoffs as explicit postcondition/precondition contracts, checked and repaired in relative silhouette space, turns an open-loop multi-agent assembly pipeline from one whose success decays geometrically with the number of stages into one whose success is roughly flat in horizon. Let $e_i$ be the relative pose error between the mating bodies at the end of stage $i$, $\epsilon$ the CAD-derived tolerance, and $S_i$ the event that stage $i$ succeeds. The goal is to replace open-loop behavior, where each stage's success depends on all earlier errors, with

$$P_{\text{total}} \approx \prod_{i=1}^{n} P\big(S_i \mid e_{i-1} \le \epsilon\big).$$

Here each factor is conditioned on a verified entry state, so it should not shrink as $i$ grows. The product rule from Instruction Agent remains. What changes is that its factors stop getting worse with depth. This is an empirical claim, not a formal guarantee, and the experiments are built to test it directly.

How This Sits in Prior Work

The evidence for the gap comes from two places. RoCo has one language-model agent per robot negotiate sub-tasks and 3D waypoints, which then seed a central motion planner. Its authors state that execution is open loop and that perception is an oracle. We keep its decomposition and add the handoff check it lacks. The RoCo Challenge at AAAI 2026 documents the resulting "sim-to-real cliff" on dual-arm gearbox assembly: a modular pipeline scored 0.256 in simulation and 0.000 on hardware because calibration offsets and occlusions propagated unchecked. That is the failure mode we target.

Multi² splits a high-level planner from a low-level executor and calls the planner only at sub-goal boundaries, which limits objective drift in text environments. We borrow its "act only at boundaries" schedule and bring it into continuous geometry. Sub-goal Distillation similarly anchors long horizons with intermediate goals, but it does not check whether those goals were actually reached.

The method draws on four pieces of rendering and verification work:

  • Soft mask gradients. Neural 3D Mesh Renderer blurs triangle edges so that mask mismatches produce useful vertex and pose gradients. We use this for rigid CAD parts.
  • Robot bodies. Differentiable Robot Rendering renders articulated robots with Gaussian splats and backpropagates image losses into joint angles. We use it to render grippers and links.
  • Collision gradients. Prof. Robot adds a signed-distance field over joint space so that pose optimization avoids self-collision and static obstacles, cutting collision rates from 99% to 17% in about 120 iterations. We use it as a regularizer during correction. The limitation it reports, that its field only holds for static scenes, constrains our design: the collision term is used only while the partner arm is holding still at the handoff.
  • Multi-view refinement. LocaliseBot refines object pose by multi-view silhouette and contour matching and reaches 99.65% grasp accuracy on OCID Grasp. It works in the absolute frame, as a one-shot pre-grasp step. We differ in two ways: our check is relative and repeated at every stage boundary.

Two verification papers shape how we handle the growth of the joint state space as arms and parts are added. ReBAR avoids this growth by moving to pairwise relative coordinates, turning an $n$-agent collision problem into $O(n^2)$ small ones. GEDC contracts verify agents locally against assume-guarantee specifications rather than checking the whole system, but those contracts are written by hand and work only on symbolic automata. We take the pairwise-relative idea from the first and the local-contract idea from the second.

Our contracts are generated automatically, from the language-model decomposition combined with CAD mating relations. They are checked from pixels, and they are statistical rather than proofs. We say this plainly and do not claim formal verification. GEDC also found that decomposition only pays off above a certain system size. We test for the same crossover.

Research Program

Contract generation. For each sub-task produced by the RoCo-style dialogue, a mating relation from the CAD assembly defines the pair of bodies involved, their target relative pose, and a tolerance $\epsilon$. For example, "gear B seated on shaft A" becomes such a contract. The handoff contract says that agent $i$'s postcondition, and agent $j$'s precondition, is that the relative pose error $e$ between the pair is at most $\epsilon$.

The check. At the boundary, wrist and near-field cameras capture the scene and a segmenter produces masks for both bodies. We parameterize the pose of body B relative to body A, render both silhouettes, and minimize an L1 mask loss plus a contour term. The checker then accepts or rejects based on the residual.

The choice of L1 is deliberate. The RoCo Challenge winners found that L2 losses give vanishing gradients near alignment in precision assembly; that finding concerned training a policy, and we test whether the same holds for silhouette losses. Because only relative pose is optimized, a constant offset in the camera extrinsics affects both bodies almost equally and mostly cancels. This is the central hypothesis, and we test it head on.

Repair and blame. If the residual is above $\epsilon$, we split the relative correction into contributions from agent $i$'s end-effector and agent $j$'s end-effector. We do this by backpropagating into each arm's joint angles through the robot renderer, with the Prof. Robot collision field as a penalty. The arm with the larger share makes a small correction, or its sub-task goes back to its own planning agent alone. The other agents' plans are left untouched. This limits the dialogue latency that RoCo reports.

Latency. Optimization runs only at handoffs, never inside the control loop. A small network trained in simulation predicts the initial relative pose, so that a few gradient steps suffice, since 120 iterations per stage would be too slow for a fast cycle. We report wall-clock time per handoff.

Calibrating the threshold. We choose the accept threshold on held-out simulated rollouts with known ground-truth relative pose, so as to hit a target false-accept rate. We report that rate on hardware separately, because it is the number that decides whether "verified entry state" means anything.

Baselines

Naive reference. The do-nothing reference is the RoCo-style modular pipeline with open-loop execution between stages: the same language-model decomposition, the same motion planner, and no handoff check. The closest reported analogue is HD-Robo in the RoCo Challenge, with 0.256 in simulation and 0.000 in the real-world track. We re-run our own open-loop version on identical hardware and demonstrations. Matching its simulation score while raising the real score clearly above zero is the minimum bar.

Strongest existing method. The strongest competitor is the end-to-end vision-language-action approach, a single network mapping camera images and instructions to motor actions. The winning ARC-VLA team reported a final score of 0.279 and 0.344 in the real-world round, with its strength concentrated in the error-recovery task. It is the strongest competitor because it absorbs sensor noise without any explicit geometry and it won on hardware. If ARC-VLA itself is not available, we train a vision-language-action policy on the same RoCo-Dataset: 300+ Isaac Sim demonstrations and 300+ teleoperated demonstrations including the failure-recovery curriculum. It gets the same cameras and a tuning budget matched to ours in GPU-hours, and we report the challenge's published numbers alongside it. Beating it means exceeding 0.344 on the real-world round under the challenge scoring, or matching it while training on no teleoperated recovery data. The second outcome would still show a data-efficiency advantage.

Absolute-frame ablation. This ablation isolates the key ingredient. It uses identical handoff timing and repair logic, but each body's pose is re-estimated in the global frame by multi-view silhouette matching in the style of LocaliseBot, and the two estimates are then compared. If this does as well as the relative check under injected extrinsic errors, our central hypothesis is false.

Other ablations. - An L2-loss variant of the relative check. - A variant that sends every failed check back to the full multi-agent dialogue instead of the blamed agent, which measures what blame assignment buys in latency and success. - A variant with the collision regularizer removed, which counts how often correction steps drive arms into contact.

Why not just…?

Why not just train a vision-language-action policy end to end? This is the real threat, since such policies won the RoCo Challenge on hardware. Their weaknesses in the file are indirect. They need curated teleoperated recovery data, they offer no explicit check that a stage succeeded, and Multi² shows flat policies degrading as horizons lengthen in its text environments. None of this proves that a policy loses on gearbox assembly. We settle the question with the matched comparison above and with the horizon-scaling experiment below. If the policy's success is as flat in stage count as ours, the handoff contracts are not justified for this task. A hybrid is also possible: the contract check could gate a learned per-stage policy, and we would test it if the policy alone looks strong.

Why not just re-localize everything at each step with a pose estimator or depth plus point-cloud registration? This is the absolute-frame ablation. The RoCo Challenge attributes the cliff to calibration offsets and occlusion, and absolute re-estimation inherits the first. LocaliseBot additionally reports that depth sensors drop out on reflective parts. The injected-extrinsic-error experiment tests directly whether relative checking is what matters or whether any re-check would do.

Why not just close the loop continuously with visual servoing, instead of only at handoffs? Continuous correction would catch errors earlier. But iterative rendering is slow, as LocaliseBot and Prof. Robot's roughly 120 iterations show, and a partner arm in motion invalidates the static collision field. We include a variant that checks at a fixed higher frequency and report success against wall-clock time. If it clearly dominates, the handoff-only schedule is the wrong choice.

Why not just add a critic agent that looks at the image and judges whether the stage succeeded? Critic roles, such as the independent tester in Self-collaboration Code Generation, catch errors a single thread misses. But a yes/no judgment gives no metric correction and no blame assignment, and sub-millimeter seating errors are unlikely to be visible to it. We include a critic that is shown the same images, and we measure its false-accept rate against ground-truth relative pose.

Proposed Experiments

Horizon scaling (the decisive test). On simulated gearbox variants built from the RoCo-Dataset assets, we vary the number of mating stages from 2 to about 10 and plot success against stage count for the open-loop pipeline, the policy, the absolute ablation, and our method. The core claim predicts geometric decay for open loop and a near-flat curve for relative contracts.

Controlled error injection. In simulation we inject camera extrinsic offsets, joint backlash, and grasp slip at known magnitudes. The pre-registered failure condition is this: if the relative check does not beat the absolute ablation under extrinsic offsets, the idea is wrong.

State-space scaling. On synthetic workcells with growing numbers of arms and parts, and on the six RoCoBench tasks, we compare the cost of pairwise contract checks against a single joint silhouette optimization over all bodies. We also locate the crossover below which the joint check is cheaper, echoing GEDC's small-system result. RoCoBench has no numeric success rates on Lacuna ("no reported result found on Lacuna for RoCo on RoCoBench"), so we re-run RoCo ourselves as the reference.

Hardware. On the Galaxea dual-arm gearbox task we report challenge score, per-handoff latency, false-accept rate of the calibrated threshold, and collisions during repair. The comparison is against the published 0.000 open-loop and 0.344 real-round policy scores, with the caveat that our hardware setup may differ from the challenge setup.