Epistemic Blame over Part Hierarchies for Repairing Robot Manipulation Failures

When a robot fails to open a drawer, a furnace door, or a sample hatch, the failure itself does not say what to fix. The segmentation model may lack knowledge of that kind of part, and a human label would help. The camera may simply not have seen the handle, and a second view would help while a label would be wasted. Or perception may have been fine and the joint stuck, so no perception change helps at all. Current systems answer "what to fix" with a confidence score. The best real-image moveable-part segmenter routes its least-confident images to human annotators (Active Coarse-to-Fine Segmentation of Moveable Parts from Real Images). Confidence cannot separate these three causes.

We propose to borrow the distinction from the formal theory of responsibility between responsibility, meaning how pivotal a decision was in the world as it actually turned out, and blame, meaning how pivotal it was expected to be given what the decision-maker could know at the time (Cause, responsibility and blame: a structural-model approach). We compute both quantities for each node of a hierarchical part decomposition of the object, from the whole object down to the door and then the handle. We then use their difference to choose the repair. High blame means the model should have known, so we ask for an annotation. High responsibility with low blame means the information was unavailable, so we move the camera. If no perception node is responsible, we charge the failure to planning or physics. The application regime that motivates this is the automated materials laboratory. There, a failed synthesis counts as evidence against a candidate material only if sample handling was not responsible, and no one currently checks.

Core Claim

Our claim is that failure-driven, hierarchy-localized blame is a better guide for spending repair effort (human labels, extra views, re-planning) than the uncertainty signals used today. The test is whether, under a fixed budget, it reaches higher manipulation success and segmentation accuracy, and whether it identifies the true failure cause more often. Two ingredients make this possible. The first is a part tree whose levels correspond to articulation rather than appearance. The second is a natural space of counterfactual alternatives at each node, obtained by cutting that tree at neighbouring heights. Together they make the responsibility computation both meaningful and cheap enough to run.

How This Differs from Existing Work

Three lines of prior work motivate the proposal. The coarse-to-fine segmenter above reaches 91.3% mask mean average precision (mAP) on real appliances with manual labels on only 11.45% of images. Its hierarchy has only two levels, object pose then part. Its queries are driven by confidence, never by interaction outcomes, and its only repair action is a human label. GREAT grounds open-vocabulary affordances in point clouds through geometric chain-of-thought reasoning. It cannot tell a knowledge failure from a blocked-clearance failure. FlowBot++ and VAT-Mart predict per-point articulation flow and action trajectories from flat point clouds. When a rollout fails, neither has a diagnostic model.

The responsibility literature supplies the method but has never touched perception. Chockler et al. define a degree of responsibility of $1/(N+1)$, where $N$ is the fewest other changes needed to make a decision pivotal. The Halpern-style account defines blame as the expectation of that degree over the agent's epistemic state. Naumov et al. separate having a preventing strategy from knowing how to use it. PATL+R makes responsibility computable in probabilistic games, but only on discrete abstract games. We reuse the Chockler and Halpern definitions directly. The new element is that the "agents" are segmentation decisions at tree nodes, and the "worlds" are scene hypotheses consistent with the robot's observations.

Hierarchical segmentation supplies the tree. SIT-HSS builds superpixel hierarchies by structural-entropy merging in 0.14 s per BSDS500 image. Ultrametric hierarchies, in the line of work by Najman, encode a tree as edge heights, so cutting at different thresholds yields nested partitions. These trees are purely photometric, so a handle with the same colour as its door merges into it. PartNet supplies the hierarchical ground truth and a warning: flat models reach only 46.4% fine-grained mean intersection-over-union (mIoU), failing precisely on thin functional parts.

Research Program

Functional ultrametric. We first build a part tree whose edge heights combine two signals: SIT-HSS-style appearance dissimilarity and disagreement in predicted articulation flow between adjacent regions, using a FlowBot++-style flow head. A handle that moves rigidly with its door but differently from the cabinet frame therefore merges with the door before the door merges with the frame, even when the colours match. The mixing weight and entropy thresholds are tuned on PartNet-Mobility. This matters because SIT-HSS is known to be sensitive to its normalisation factor and entropy threshold across imaging domains.

Responsibility and blame. We then define both quantities per node. Let $v$ be a node on the tree path to the manipulated part, $s_v$ its committed segmentation, and $\mathcal{A}_v$ its alternatives: the partitions obtained by cutting the ultrametric one level above or below at $v$. Let $w$ be a world, here a full scene configuration, and $Y(s, w)\in\{0,1\}$ the success of re-executing the plan in simulation under segmentation $s$. Following Chockler and Halpern, the responsibility $\rho_v(w)$ is $1/(N_v(w)+1)$. Here $N_v(w)$ is the minimum number of other nodes whose segmentations must be changed before swapping $s_v$ for some alternative in $\mathcal{A}_v$ flips $Y$. Blame then averages responsibility over the robot's belief:

$$ \beta_v = \mathbb{E}_{w \sim P(w \mid o)}\big[\rho_v(w)\big], \qquad g_v = \rho_v(w^\star) - \beta_v . $$

In this equation, $P(w\mid o)$ is the robot's posterior over scenes given its observations $o$. We approximate it by sampling completions of occluded geometry and joint parameters consistent with the depth image. $w^\star$ is the true world. It is available in simulation, and in deployment it is approximated by the post-failure observation, which typically reveals what was hidden. The quantity $g_v$ is the observational gap.

Repair rule. The rule follows directly. If some node has large $\beta_v$, we request a human label at that node's scale; the label can be a single handle mask rather than a full image. If $g_v$ dominates instead, we move the camera to the view that maximally reduces posterior entropy over $v$. If $\rho_v$ is near zero for every perception node, we attribute the failure to planning or physics and route it to re-planning, or in the lab setting to the chemist.

Cost control. The minimum-$N$ search is restricted to the root-to-target path, which is at most about five nodes deep, and to a handful of belief samples. Counterfactual rollouts therefore stay in the tens per failure.

Laboratory pilot. As a constrained pilot of the validation-gap use case, we model articulated lab fixtures (furnace doors, sample drawers, crucible trays) as additional simulation assets. We then ask whether handling responsibility can be separated from non-handling causes. The automated lab in A Synthesizability-Guided Pipeline for Materials Discovery reports 9 failures out of 16 targets with no diagnosis. Its classifier is also trained under positive-unlabeled bias, where unsynthesised compounds are treated as negatives. A "handling not responsible" certificate is exactly what would be needed before treating a failed run as a true negative. We do not claim to attribute chemical or kinetic causes.

Baselines

Uniform repair (naive reference). Every failure triggers a human annotation of the full image, with no view changes. This shows what an unrouted budget buys.

Confidence-driven active learning. This is the coarse-to-fine active-learning loop from Active Coarse-to-Fine Segmentation of Moveable Parts from Real Images, using its confidence-based query selection. It is the strongest existing method on real moveable-part segmentation: 91.3% mAP with 11.45% of images manually labeled, against 68.4% for OPDFormer-C and 23.1% for Grounded-SAM, as reported in that paper. To keep the comparison fair, we give it the same segmentation backbone, the same total budget (counting one label and one view change at cost ratios we fix in advance), and the same rounds. We also give it a view-change action triggered by low confidence, so that the comparison isolates the decision signal rather than the action set. Beating it means higher mAP at equal budget on the real appliance set, or equal mAP at a lower budget, with non-overlapping confidence intervals over seeds.

Ensemble disagreement. A deep ensemble splits uncertainty into a disagreement (epistemic) part and a residual (aleatoric) part. High disagreement routes to labels and high residual routes to new views. We found no reported result on Lacuna for this routing on articulated-part benchmarks, so we will run it ourselves with an ensemble whose compute matches our counterfactual rollouts.

Responsibility-only ablation. This uses $\rho_v(w^\star)$ alone, i.e. oracle-replay attribution with no epistemic weighting. It isolates whether blame adds anything beyond "which node's error flipped the outcome."

Flat-tree ablation. This computes blame over a single-level partition. It isolates the hierarchy.

Appearance-only ablation. This builds the tree from SIT-HSS alone, with no flow term. It isolates the functional ultrametric.

Manipulation performance references. For downstream manipulation we report VAT-Mart's 55.14% door-pushing success in SAPIEN and FlowBot++'s 0.10 normalized distance to goal on unseen PartNet-Mobility categories. These serve as the base policies that repair is layered on, not as competitors.

Why not just…?

Why not just use the segmenter's confidence to decide what to fix? Confidence is computed before acting and is blind to the outcome. An occluded handle can be segmented confidently but wrongly, and a confidently correct mask can still be followed by a stuck joint. The coarse-to-fine system's 60% reduction in human effort shows routing matters, but it routes everything to humans. The view-augmented confidence baseline above tests directly whether outcome-driven blame beats outcome-blind confidence once both have the same actions.

Why not just separate epistemic from aleatoric uncertainty with an ensemble? This is the most serious threat. Ensemble disagreement does capture "the model doesn't know." But occlusion is neither model ignorance nor irreducible noise. It is missing information that an action can supply. Ensembles may agree confidently on a hallucinated completion, and they say nothing about whether the error mattered for the outcome. If ensemble routing matches our attribution accuracy on the occlusion-injected split, the epistemic-weighting component is not earning its cost, and we would simplify the method to ensemble routing plus outcome replay.

Why not just replay the failure with each stage swapped for ground truth? In simulation this is essentially actual responsibility, and it is our responsibility-only ablation. It has two limits. It needs ground truth that deployment lacks. More importantly, it cannot say whether the robot could have done better from its own view, which is the question that separates "label it" from "look again." If the ablation matches full blame on repair efficiency, the epistemic part of the claim fails.

Why not just train an RL policy with failure penalties? VAT-Mart documents mode collapse in plain RL exploration on articulated objects. Even a successful policy returns a scalar reward, not a diagnosis, so it cannot tell a chemist whether a failed run is evidence about the material.

Proposed Experiments

Simulated failure-cause study. The decisive test runs in SAPIEN on PartNet-Mobility, using the FlowBot++ category split (11 training classes, 10 unseen). We inject failures whose cause is known by construction: occluders and grazing camera angles (observational), held-out categories and rare handle geometries (knowledge), and raised joint friction or locked joints (physical). The primary metric is macro-F1 in recovering the injected cause. The secondary metric is manipulation success and normalized distance to goal after a fixed repair budget. The prediction is that full blame beats ensemble routing and the responsibility-only ablation mainly on the occlusion split, and that the two ablations tie with it on the physics split.

Hierarchy quality. On PartNet hierarchical segmentation, we check that the functional ultrametric improves fine-grained part mIoU over appearance-only trees. The reference point is PointCNN's 46.4%. On BSDS500 we check that it does not degrade boundary recall relative to SIT-HSS.

Real-image transfer. On the real articulated-appliance dataset and OPDReal, we rerun the active loop at matched budget against the 91.3% / 11.45% reference. Here views are re-sampled from the dataset's multiple images per object rather than physically moved.

Laboratory pilot. Finally, we run lab-fixture assets with injected handling faults versus clean handling. We report how often "handling not responsible" is certified correctly, since a wrong certificate would mislabel a material as unsynthesizable.

Kill criteria. The project is falsified if blame fails to beat ensemble routing on attribution F1 under occlusion. It is also falsified if the functional tree does not beat the flat-tree ablation on repair efficiency.