A robot that opens drawers, lifts lids and turns knobs on objects it has never seen usually works by breaking each object into functional parts (a handle, a lid, a knob) and predicting where and how to act on each one. These systems generalize surprisingly well across object categories, but they fail often. The best part-aware policy in one recent study succeeds on 28.79% of simulated generalization tasks and 58.75% of unseen real objects (PA3FF). Every failure is a physical event that can damage the robot or the object. Selective prediction is the discipline of letting a model say "I won't act on this one" while guaranteeing a bounded error rate on the cases it does act on. It offers exactly the missing tool. Its guarantees, however, rest on one assumption: the test cases must be statistically interchangeable with the held-out cases used to set the abstention threshold. Unseen object categories break that assumption.
This proposal tests whether the premise that makes part-based affordance models generalize also restores the statistical assumption. That premise is that a handle is a handle, whether on a fridge or a microwave. If local functional geometry really is shared across categories, then calibration errors grouped by functional part class should be close to interchangeable across object categories, even when errors grouped by object are not. In that case, abstention thresholds set per part class on seen categories should keep their risk guarantees on unseen categories. The practical result would be a pre-execution gate. Before the arm moves, it decides for each (part, action) pair whether to act or defer, and it certifies the failure rate of that decision separately for each part class, including rare ones.
Core Claim
We claim three things. First, per-part-class selective risk control on execution failure keeps its nominal guarantee under held-out-category shift, while pooled (single-threshold) control does not: pooled control will over-admit rare, hard part classes. Second, a part-conditioned confidence score does better than the detector's own probability, and it costs only one forward pass. The score combines detection confidence with the spread among non-equivalent pose hypotheses that a symmetry-aware pose estimator already produces. Third, this pre-execution gate prevents failures that runtime monitors only catch after contact.
The first claim is falsifiable. If pooled control with a good score already holds its bound on unseen categories, the stratification contribution is wrong, and we will report that.
Relation to Prior Work
Two lines of robot work touch this intersection, and both act at a different time than we do. Uncertainty-Aware Failure Detection for Imitation Learning Robot Policies learns scalar uncertainty signals from successful rollouts. It then wraps them in time-varying conformal bands (conformal prediction: a distribution-free method for turning scores into guaranteed error rates) to flag failures during execution. SAFE probes the internal features of vision-language-action controllers with a sub-millisecond detector. Both monitor monolithic policies at runtime. Neither knows about parts. Neither can refuse before the gripper touches something. We reuse their low-latency philosophy but move the decision before execution and attach it to the affordance itself.
Uncertainty-driven Affordance Discovery uses ensemble disagreement over affordance maps as an exploration bonus, actively seeking the uncertain actions. We use uncertainty in the opposite direction, to avoid acting, and we add a coverage–risk guarantee that the exploration setting has no need for. Selective Contrastive Learning for Weakly Supervised Affordance Grounding is "selective" only in the sense of switching training objectives between part and object cues. It has no inference-time abstention.
On the selective-prediction side, three works supply our method and one supplies the motivation. Selective Classification for Deep Neural Networks gives the Selection with Guaranteed Risk procedure. This procedure uses binomial bounds on a held-out set to certify that selective risk stays below a target $r^*$ with confidence $1-\delta$. That paper also found raw maximum softmax probability to be a strong ranking signal, which makes it our main competitor rather than a straw man.
Act or Defer motivates stratification. It shows that population-level conformal control can collapse on rare classes: roughly 10% selected coverage with 90% error on pancreatic cancer at nominal $\alpha=0.10$, where stratified control maintained over 99% coverage. Its setting is clinical and assumes no distribution shift. Our question is whether a stratification chosen for physical reasons survives a category shift. Towards Improving Selective Prediction Ability of NLP Systems shows that a secondary calibrator built on instance-difficulty features beats MaxProb out of domain. That result justifies learning a small scorer rather than trusting the detector's own probability.
Finally, PAC Bench constrains the design. It finds that foundation models reach near 0% on multi-label affordances and at most 25.8% on physical constraints. This rules out a vision-language model as the verifier, and it directly questions the frozen-model part grounding that AdaRPG relies on.
Research Program
The substrate is GAPartNet. It annotates 9 generalizable part classes across 27 categories. Each part carries a canonical pose whose axes align with the interaction direction, which means every part prediction already implies an action primitive (pull along z, rotate about an axis, press). Each prediction is therefore an affordance triad of part, action and expected effect. We label it with a binary execution loss: whether the scripted part-conditioned manipulation in the SAPIEN simulator achieves the effect, meaning the drawer displaced or the lid lifted past a threshold. This matters because selective risk on execution, not on segmentation overlap, is what protects hardware. We also record perception proxy losses, detection IoU below 0.5 or rotation error above a fixed tolerance. These let us study how perception confidence relates to execution outcome.
The confidence score is computed once per detected part. It combines three inputs: the segmentation/detection confidence; the dispersion among the 64 quaternion candidates produced by the symmetry-aware estimator of SAFAG, measured after collapsing hypotheses the estimator deems symmetry-equivalent; and simple point-support statistics such as point count and depth-hole fraction on the part, which target the reflective-surface failures GAPartNet reports. A shallow calibrator, following the secondary-calibrator idea of Towards Improving Selective Prediction Ability of NLP Systems, maps these inputs to a failure score. The calibrator is trained with category-level cross-fitting, so no category used for training the scorer is used for calibration.
The gate is stratified by part class $k$. For each $k$, let $\hat{s}$ be the failure score and $\tau_k$ a threshold, so that the robot acts when $\hat{s} \le \tau_k$. Selective risk for class $k$ is
$$R_k(\tau_k) = \Pr\big(\text{failure} \mid \hat{s} \le \tau_k,\ \text{part class} = k\big).$$
On calibration data from seen categories, we choose the largest $\tau_k$ whose binomial upper confidence bound on $R_k$ is at most $r^*$, at level $\delta/K$ across the $K$ classes. The coverage-maximizing search follows Geifman and El-Yaniv's procedure, applied per stratum. The guarantee then reads: with probability at least $1-\delta$, every part class simultaneously has execution risk at most $r^*$. This holds provided that, within a part class, unseen-category instances are exchangeable with seen-category ones.
That proviso is the empirical hypothesis, and we test it directly. For rare classes with too little calibration data, the bound yields zero coverage. We accept that honestly. The gate defers such parts rather than borrowing strength, and the lost coverage becomes an output of the study.
Baselines
The naive reference executes every predicted part. Its coverage is 100% and its risk equals the base failure rate on unseen categories. GAPartNet reports 90–95% manipulation success in SAPIEN, largely on seen objects, and only 37.2% AP50 segmentation on unseen categories. This suggests a much higher unseen-category failure rate, which we will measure directly.
The strongest existing competitor is pooled Selection with Guaranteed Risk using the detector's softmax or detection confidence, as in Selective Classification for Deep Neural Networks. It is strongest because that paper found this simple score competitive with Monte Carlo dropout, and because the pooled bound is itself valid under exchangeability. The only published numbers for this method are on image classification: 1% error at 78% coverage on CIFAR-10, and 2% top-5 error at about 60% coverage on ImageNet. We found no reported result on Lacuna for any selective-prediction method on GAPartNet or any part-affordance benchmark, so we will establish this baseline ourselves.
We give it the same perception backbone, the same calibration data and the same $(r^*, \delta)$. Beating it means one of two things. Either our gate keeps every part class at or below $r^*$ on held-out categories in at least $1-\delta$ of resampled splits while the pooled gate violates the bound on some class, or both are valid and ours reaches higher total coverage.
A second stratified competitor adapts the class-stratified control of Act or Defer, but stratifies by object category. Unseen categories have no calibration data, so this baseline must fall back to pooled thresholds. It isolates whether the stratification axis matters, as opposed to stratification itself.
Two ablations isolate our ingredients. The first swaps our learned score for raw detection confidence inside the part-stratified gate. The second holds the score fixed and removes stratification.
A compute-matched deep ensemble of five perception models, scored by disagreement, tests whether sampling-based uncertainty justifies its cost.
A runtime monitor is trained in the style of Uncertainty-Aware Failure Detection for Imitation Learning Robot Policies: a random-network-distillation signal with functional conformal bands on the same rollouts. It is compared on failures stopped before contact.
Finally, a GPT-4o verifier is prompted with the image and the proposed part and action, following AdaRPG-style prompts, and asked whether execution will succeed.
Why not just…?
Why not just threshold the detector's confidence? This is the real threat, and the proposal is built so that it can win. Geifman and El-Yaniv found softmax response strong within distribution. Against it, Towards Improving Selective Prediction Ability of NLP Systems reports that MaxProb degrades out of domain, with a 5.64% area-under-risk-coverage gain for a secondary calibrator on out-of-domain inference data. An arbitrary threshold carries no guarantee at all. The raw-confidence ablation inside the stratified gate settles the question. If it matches our learned score in both coverage and bound validity on held-out categories, we drop the learned score and keep the simpler gate.
Why not just use Monte Carlo dropout or an ensemble? Sampling multiplies inference cost. Both SAFE and the imitation-learning failure detector, whose RND signal runs 36–44x faster than sampling-based STAC, argue that cheap single-pass signals suffice. Our pose-hypothesis spread is multi-hypothesis information the estimator already computes for free. The compute-matched ensemble baseline measures whether the extra passes buy coverage. If they do, we report the latency–coverage trade rather than hide it.
Why not ask a vision-language model whether the action is safe? PAC Bench reports 0% multi-label affordance accuracy and at most 25.8% constraint accuracy for leading models. Even a perfect verbal verifier would come with no error guarantee. We include the GPT-4o verifier so that its risk–coverage point can be read off directly, rather than argued.
Why not just stop the robot when a runtime monitor fires? Runtime detection works after contact, and it is imperfect: SAFE reaches 64.16% ROC-AUC on real Franka tasks. It is complementary to our gate, not a substitute. We will measure the fraction of failures each method catches before any contact force, and whether stacking the gate in front of the monitor reduces total failures relative to the monitor alone.
Why not stratify by object category, or just pool? Object-category thresholds cannot exist for categories the robot has never seen. Pooling is valid only on average, and Act or Defer shows how badly averages can hide rare-class collapse. The comparison against these two alternatives is the core experiment.
Proposed Experiments
The decisive test is leave-categories-out on GAPartNet's seen/unseen split, repeated over many random calibration/test resamplings at $r^* \in \{0.05, 0.10\}$ and $\delta = 0.1$. It reports the frequency with which any part class exceeds $r^*$ on unseen categories, the coverage per part class, and the total area under the risk–coverage curve.
The hypothesis predicts two things. The pooled gate should violate the bound on rare classes such as slider buttons more often than $\delta$. The part-stratified gate should stay within $\delta$.
A direct exchangeability diagnostic accompanies this test. A two-sample test on score–loss pairs compares seen and unseen categories within each part class against the same test on object-level groupings. If part-level groupings are no more exchangeable than object-level ones, the central claim fails regardless of downstream numbers.
A second experiment probes shift the stratification cannot absorb. It transfers thresholds calibrated on GAPartNet to PartNet-Mobility categories used by AdaRPG, and adds synthetic depth noise on reflective parts. The purpose is to map where the guarantee breaks.
A small real-robot study checks whether simulated bounds track physical failure rates, using the drawer, microwave, bottle and lamp settings in which AdaRPG reports 8/10 to 10/10 successes. It is sized honestly: tens of trials per part class can check consistency, not certify a bound.
Across all experiments, the result that would most change the field's practice is a clean one: calibration keyed to functional parts transfers across object categories, and calibration keyed to objects does not.
References
- PA3FF:Learning Part-Aware Dense 3D Feature Field For Generalizable Articulated Object Manipulation
- Uncertainty-Aware Failure Detection for Imitation Learning Robot Policies
- SAFE: Multitask Failure Detection for Vision-Language-Action Models
- Uncertainty-driven Affordance Discovery for Efficient Robotics Manipulation
- Selective Contrastive Learning for Weakly Supervised Affordance Grounding
- Selective Classification for Deep Neural Networks
- Act or Defer: Error-Controlled Decision Policies for EHR Foundation Models
- Towards Improving Selective Prediction Ability of NLP Systems
- PAC Bench: Do Foundation Models Understand Prerequisites for Executing Manipulation Policies?
- Adaptive Articulated Object Manipulation On The Fly with Foundation Model Reasoning and Part Grounding
- GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable Parts
- Generalizable and Actionable Parts Pose Estimation with Symmetry Annotation-Free Learning Strategy