Kinematic-Consistency Gating for Part Re-Detection in Articulated Manipulation

Robots that open drawers, doors and lids increasingly work in two stages. First they find the functional part, such as a handle, knob or lid, using large vision models. Then they run a short sequence of discrete skills such as "grasp", "pull" and "rotate" on that part. The weak point is what happens between those two stages. Once execution starts, nothing checks that the part being tracked is still the right part, or that it is moving the way the plan assumed. The authors of PartInstruct name part grounding, meaning keeping hold of the right part as the object rotates, opens and becomes occluded, as the main source of failure. Their bi-level system reaches 31.7% success, against 15.4% for an end-to-end policy (PartInstruct). Tracking research has a standard fix for lost targets, called confidence-triggered re-detection: run a cheap local tracker, and wake an expensive global detector only when a confidence score says the track has failed. Its confidence scores, however, are built on appearance, and an articulated part changes appearance precisely when things are going well. A handle seen edge-on as the door swings looks like a tracking failure.

We propose to gate re-detection on kinematic consistency rather than on appearance. Before acting, the robot estimates the part's joint: hinge or slider, axis, and origin. That joint model, together with the robot's own commanded gripper motion, predicts where the part should be at every control step. The trigger signal is the gap between that prediction and what the tracker observes. The gap is measured in millimetres and degrees, so its threshold can be set from known estimation noise rather than tuned on an offline validation set. Its shape also distinguishes three different failures: the part is lost, the gripper has slipped, or the joint model was wrong. The discrete skill library then plays a useful role. Because actions are a finite set of skills, recovery becomes a small choice of which skill to re-enter with which corrected part estimate, rather than a continuous replanning problem.

Core Claim

The claim is that a part tracker's failures during articulation are best detected by checking the expected effect of the action, not the appearance of the part. The effect is the "resulting effect" leg of the affordance relation ⟨part, action, effect⟩ described in Building Affordance Relations for Robotic Agents - A Review. A residual from that check should do three things better than appearance-based critics. It should raise fewer false alarms while the part legitimately changes aspect. It should catch slips that appearance cannot see, because a slipped handle still looks exactly like the handle. And it should disambiguate identical neighbouring parts during re-detection: of two identical drawer handles, only the one on the predicted arc or slide line is consistent with the joint. If this holds, it closes the specific open-loop gap that CLAP's authors acknowledge (CLAP) and that 3D-LOTUS++ shares (3D-LOTUS++). It does so without running heavy detection every frame.

Relation to Prior Work

The proposal reuses its re-detection architecture from tracking. STORM pairs a fast 6D pose tracker with a learned compatibility critic that compares the current crop against a memory of past successful frames. When the critic's energy-like mismatch score stays high, STORM wakes a global segmentation module. This reaches 74.0 ADD-AUC on LineMOD-Occluded at 25–33 frames per second (STORM). STORM handles rigid whole objects, its thresholds come from offline calibration, and it never drives a robot. We keep its two-stage, mostly dormant structure and replace the appearance critic with a kinematic one.

Two other tracking works motivate the specific failure we target. RTracker validates re-detection candidates against a memory tree of target and distractor appearances (RTracker). The file notes that such memories misclassify a target whose geometry changes quickly. An articulating part is exactly such a target. Gard et al. use an edge-matching score to trigger global re-initialization for similar-looking objects (Combining Local and Global Pose Estimation for Precise Tracking of Similar Objects). That score is again appearance-based.

The joint model comes from the affordance literature. GMAP predicts joint type, axis, origin and motion range from point clouds, with 0.42° axis-orientation error (GMAP). GAPartNet defines part coordinate frames aligned with the pulling direction (GAPartNet). SAFAG handles symmetric parts such as knobs by keeping many orientation hypotheses (SAFAG). We use these as the predictive model inside the gate, which none of them does: each estimates the joint once and then executes open-loop.

The closest work that combines confidence with parts is the frame-consistent uncertainty sampling in Part-Guided 3D RL. It down-weights flickering part points but never suspends execution or re-detects. AdaRPG grounds parts with GPT-4o, GroundingDINO and SAM, then runs scripted skills, also without runtime checks (AdaRPG). Its ablation without exact part prompts fails badly. That result constrains our design: re-detection must not depend on re-prompting a foundation model from scratch, which is one reason we constrain the re-detection search geometrically.

Research Program

The system has three pieces wrapped around an existing bi-level manipulation stack: a vision-language planner that emits discrete part-level skills, a grounding module that produces part masks, and a low-level policy.

The first piece is the predictor. At grounding time, a joint estimator in the style of GMAP outputs a joint model $J$: its type, axis $a$, origin $o$, and range. When a skill $k$ begins, we store the part's pose $T_0$. While the gripper holds the part, the predicted part pose $\hat T_t$ is the gripper's commanded displacement projected onto the one-degree-of-freedom motion that $J$ allows: a rotation about $a$ through $o$ for a hinge, a translation along $a$ for a slider. Before contact, the prediction is simply that the part does not move.

The second piece is the residual gate. A local tracker, a SAM2 mask lifted to a part point cloud and fitted against the reference part, gives an observed pose $T_t$. We compute two normalized residuals. The first is $r^{p}_t = d(T_t, \hat T_t)/\sigma_J$, the gap between observed and predicted part pose, where $d$ combines translation and rotation error and $\sigma_J$ is the expected joint-estimation error. The second is $r^{g}_t = d(T_t, G_t)/\sigma_g$, the drift between the part and the gripper pose $G_t$ while grasped, where $\sigma_g$ is the expected grasp-pose noise. A failure is declared when a cumulative-sum statistic exceeds a threshold $h$:

$$ S_t = \max\big(0,\; S_{t-1} + r_t - \kappa\big), \qquad \text{trigger when } S_t > h, $$

where $r_t$ is either residual and $\kappa$ is the residual level tolerated as normal noise. A cumulative sum suits this problem because slips and drift accumulate gradually rather than appearing as single-frame spikes. Because both residuals are scaled by physical noise levels, $\kappa$ and $h$ have a direct meaning, for example "two standard deviations of joint error sustained for 200 ms". We test whether one setting transfers across object categories without per-category calibration. That transfer is the online-threshold problem STORM leaves open.

The third piece is diagnosis and re-entry. The pattern of residuals, together with tracker visibility, sorts each trigger into one of three cases. In the first, the mask has vanished or collapsed, so the part is lost. We then run global re-detection with GroundingDINO and SAM, restricted to candidates whose 3D position lies within the joint's reachable motion. This rejects identical handles elsewhere on the object. In the second, the part is static while the gripper moves, which indicates a slip; we re-enter the "grasp" skill at the current part pose. In the third, the part moves coherently with the gripper but off the predicted path, so the joint model is wrong. We re-fit the axis to the observed part trajectory, which is now informative, and resume the current skill. In every case, recovery is a choice from the finite skill graph. That choice is where the discrete action space helps rather than hurts.

Baselines

The naive reference is the unmodified bi-level stack with open-loop SAM2 tracking. This is the PartInstruct setting, whose reported number is 31.7% success for Gemini-2.0 Flash with DP3-S across the five generalization test sets (PartInstruct). We re-run it ourselves so that every comparison uses the same seeds and splits.

The strongest simple competitor is per-skill re-detection: re-run full GroundingDINO and SAM grounding at the start of every discrete skill. This is what a practitioner would try first. With keyframe-style actions it is cheap, and it fixes failures that happen between skills.

The strongest existing gating method is a STORM-style appearance critic retrained on part crops. It compares current part crops against a memory of earlier frames and uses the same cumulative-sum trigger and the same recovery module as our system, so it differs from ours only in the confidence signal. We also include the confidence score SAM2 itself produces, thresholded, as the cheapest gate, and an upper bound that re-detects on every frame.

Ablations isolate the key ingredients. The first keeps our trigger but always re-detects, with no diagnosis. The second removes the joint-reachability constraint from re-detection. The third uses only the part-pose residual, and the fourth only the part-gripper drift. An oracle trigger, which fires from simulator ground truth, bounds the achievable gain.

All methods share the grounding models, low-level policy, demonstrations, and a single tuning budget: thresholds are set on one held-out split for every gate. Because more detection can always buy success, we also report success as a function of foundation-model calls per episode and of wall-clock latency. Beating per-skill re-detection means higher success at equal or fewer detector calls, or equal success with substantially fewer false or missed recoveries in the middle of skills. Beating the appearance critic means a lower false-alarm rate during correct articulation at a matched rate of missed failures.

Why not just…?

Why not just re-detect at the start of every skill? This is the real threat. In simulation, where compute is free, it may match us on overall success for tasks where failures happen between skills. Two things should separate the methods. First, failures in the middle of a skill, such as a slip during a pull, happen after the check and are invisible to it. Second, repeated unconstrained re-detection risks switching to an identical neighbouring part, a risk STORM addresses with language prompts for identical distractors and AdaRPG shows to be prompt-sensitive. Our perturbation suite separates mid-skill slips from between-skill displacements. If per-skill re-detection matches us on both at equal call budgets, we will report that the kinematic gate's value is limited to latency and compute savings.

Why not just use an appearance-based critic, as STORM does? STORM's critic was validated on rigid objects (STORM). The file records the expected failure mode for appearance memories when target geometry changes rapidly, as noted for RTracker. An appearance critic should also be blind to slips. The matched comparison in our experiments, identical except for the confidence signal, tests both predictions directly. If the appearance critic does as well, the kinematic predictor is unnecessary and we will say so.

Why not just run the detector every frame? STORM's real-time rate depends on keeping its global module dormant. Per-frame detection also discards temporal continuity and makes part poses jitter, which the low-level policy then has to absorb. We include it as an upper bound on detection coverage and report its latency. On a real robot, latency is what matters.

Why not just train a stronger end-to-end policy? On PartInstruct, the end-to-end DP3 policy reaches 15.4% against 31.7% for the decomposed system. On GemBench's long-horizon level, end-to-end models score 0% where 3D-LOTUS++ reaches 17.4% (3D-LOTUS++). Smoother continuous action outputs such as NIAF remove jitter, but by their authors' own account they do not provide recovery from failed plans. Our gate is orthogonal to the choice of low-level policy, and could interrupt such a continuous policy just as easily.

Proposed Experiments

The main testbed is PartGym from PartInstruct. Its five generalization splits are novel positions, instances, part combinations, task categories and unseen objects, and the benchmark's documented bottleneck is the one we target. The no-gate bi-level baseline's reported 31.7% is the reference to beat. PADP's 28.79% from PA3FF is an additional point of comparison, though it is reported jointly over PartInstruct and RLBench.

The second testbed is GemBench, using the level built around novel articulated objects and the long-horizon level. We wrap 3D-LOTUS++ and, where code permits, CLAP. CLAP reports 62.0% average and 31.4% on the long-horizon level (CLAP). No reported result was found on Lacuna for either method on the articulated-object level alone, so we measure it ourselves.

Because natural benchmark episodes contain few failures in the middle of a skill, we add a controlled perturbation suite on PartNet-Mobility and AdaManip objects. It injects four kinds of event at random times: forced gripper slip, occlusion of the handle by the arm, displacement of the whole object, and a duplicate identical handle placed nearby. For failure detection alone, we will also report results on STORM's Tracking Failure Benchmark, for which no reported number was found on Lacuna. That benchmark contains only rigid objects, so it checks that the gate does not lose ground there rather than showing its advantage.

The metrics are task success; detection latency and false-alarm rate per perturbation type; the rate at which each trigger is diagnosed correctly; recovery success conditioned on a correct trigger; the rate of switching to the wrong part after re-detection; and detector calls and latency per episode.

Three checks decide the outcome. On the duplicate-handle and occlusion perturbations, joint-constrained re-detection should cut wrong-part switches compared with unconstrained re-detection. On mid-skill slips, our gate should detect and recover where per-skill re-detection cannot. Most importantly, during unperturbed articulation the appearance critic should false-alarm noticeably more often than the kinematic gate at a matched miss rate. If that last result fails, the central claim fails.

A modest real-robot replication on drawers and cabinet doors with deliberately induced slips will test whether thresholds set from physical noise levels transfer without re-tuning.