Certified Ask-or-Act Frontiers for Ambiguous Robot Instructions Under User Shift

When a person tells a robot "hand me the mug" and there are three mugs on the table, the robot has two choices. It can ask a question, which costs the person time and patience, or it can act, which risks grabbing the wrong object. Existing systems make this choice with a fixed threshold. That threshold is tuned on one population of instructions and then trusted on everyone else. This proposal builds a calibration procedure that shows a robot operator the whole trade-off curve between "wrong objects grabbed" and "questions asked." Each point on the curve carries a finite-sample guarantee. The guarantee is designed to survive the shift that matters most in practice: a new user, a new room, or a vaguer speaking style than the robot was calibrated on.

The construction combines two tools. The first is inverse conformal risk control. Conformal methods are distribution-free statistical procedures that turn held-out data into guaranteed error bounds; the inverse variant certifies a whole curve of operating points instead of a single target. The second is a robustness buffer whose size is itself set by a conformal argument over users, not by a loose theoretical bound or by hand-tuning. In two sentences: treat the set of plausible referents as an uncertainty set indexed by a threshold, and certify the risk-versus-questions curve for that whole family. Then add a buffer, calibrated across held-out users or scenes, so that the certified curve still holds for a new user with high probability.

Core Claim

Two results motivate this work. First, ambiguity sets built from language-model token probabilities do not match real ambiguity. On the AmbiK kitchen benchmark, which evaluates conformal methods such as KnowNo and LAP alongside binary prompting, no method reached 20% set size correctness, and almost all scored under 10% at telling ambiguous tasks from their unambiguous twins. Second, the calibration that does exist assumes calibration and deployment instructions are exchangeable. KNOWNO converts an LLM planner's multiple-choice probabilities into conformal sets and asks for help when the set is not a singleton. It guarantees a user-chosen success rate such as 85%, but only under that exchangeability. REI-Bench shows how badly this can fail: planner success drops by up to 36.9% when references move from explicit to implicit.

We claim three things. First, referential grounding is an unusually clean setting for inverse conformal risk control. Its losses are naturally bounded: acting on a wrong referent costs 1, and dialogue length is capped by the candidate count. That bound is exactly the condition CREME requires and finds hard to meet in open-ended problems. Second, a buffer added to the risk scale and sized by a conformal quantile over exchangeable users or scenes gives a frontier that remains valid on new users. Third, both the frontier and the buffer stay tight only if the per-candidate score comes from grounded signals, such as cross-modal attention mass on each candidate, rather than from token probabilities.

Relation to Prior Work

Two lines of work on ambiguity-aware robots supply the setting but not the calibration. CLUE observes that text-to-image attention in a middle layer of PaliGemma 2 splits across candidates when an instruction is ambiguous. A small convolutional network on these maps reaches 0.846 F1 for ambiguity detection, against 0.683 for prompting. Its decision threshold, however, is uncalibrated and carries no risk guarantee. AmbiRefer3D generates candidate sets in 3D scenes and asks questions chosen to split the remaining set. This raises grounding accuracy from 23.51% to 41.57% with 3.39 rounds on average, but the decision of when to stop asking is heuristic. Ask-to-Clarify emits an ambiguity token from a vision-language model and hands clarified commands to a diffusion policy; nothing bounds the cost of a wrongly emitted "act" token. LEGS-POMDP fuses language and pointing in a Bayesian belief with fixed observation models. It is a natural future host for our calibrated stopping rule but is not our target here.

The method itself comes from the calibration literature. CREME supplies the inverse conformal risk control estimator and the data-splitting step that keeps an operator's post-hoc choice of operating point valid. Its experiments cover only linear programs, newsvendor, portfolio, and shortest-path problems. A Unified Perspective on Conformal Prediction and Wasserstein DRO shows that conformal prediction corrects the quantile level while distributionally robust optimization (DRO) adds a buffer in score value. Under Lévy-Prokhorov shift, which combines mass displacement with local perturbation, the right correction combines both.

That paper also constrains our design in two ways. It warns that radii certified by concentration inequalities are too loose to use, and that tuning the radius empirically voids the guarantee. It also reports that conformal sets on discrete labels can jump straight to the full label set. Our group-level conformal buffer is meant to get past the radius dilemma: the buffer is tuned from data, yet it keeps a guarantee at the level of users.

WR-CP handles covariate and concept shift with importance weights plus a Wasserstein penalty in training. It reports a 3.2% coverage gap and sets 37% smaller than worst-case conformal baselines. The penalty requires retraining the model, which rules it out for frozen foundation models. We use it only as a reference point.

Research Program

Each test instance contains an instruction, a scene, a list of candidate objects, and a set $V$ of objects that satisfy the speaker's intent. $V$ may contain more than one object when several would genuinely do. For each candidate $j$ we compute a score $s_j$. Our default is the fraction of middle-layer attention mass that falls inside candidate $j$'s segmentation mask, taken from the same PaliGemma 2 layer CLUE uses. For 3D scenes we use the candidate scores of the D-LISA grounder that AmbiRefer3D builds on. The candidate set at threshold $\lambda$ is

$$C_\lambda = \{ j : s_j \ge \lambda \cdot \max_k s_k \},$$

that is, every candidate whose score is within a factor $\lambda$ of the best. If $C_\lambda$ is a single object, the robot acts. Otherwise it asks set-splitting questions in the style of AmbiRefer3D until one object remains, with answers simulated from each dataset's dialogue annotations.

Two losses are recorded for each instance. The risk is $L^{\text{err}}_\lambda = 1$ if the robot finally acts on an object outside $V$, and 0 otherwise. The cost $L^{\text{ask}}_\lambda$ is the number of questions asked. CREME applied to these bounded losses yields a certified upper curve $\bar R(\lambda)$ on expected risk across all $\lambda$, paired with an estimated question cost.

For shift, calibration data are grouped into units we treat as exchangeable: scenes in AmbiRefer3D, image sources in InViG, and users or sessions in a small hardware study. For each held-out group $g$, we measure how far that group's risk exceeds the pooled certified curve at its worst threshold:

$$\Delta_g = \max_\lambda \big(\hat R_g(\lambda) - \bar R(\lambda)\big),$$

where $\hat R_g$ is the group's empirical risk. The buffer $\rho$ is the $\lceil (G+1)(1-\delta)\rceil$-th smallest of the $G$ group values, the standard split-conformal quantile. The deployed frontier is $\bar R(\lambda) + \rho$. Under exchangeability of groups, a new group's risk curve lies below this frontier with probability at least $1-\delta$, up to the finite-sample noise in each $\hat R_g$. We will bound that noise term explicitly, or remove it by requiring a minimum group size.

This is the level-plus-value combination that the Unified Perspective paper recommends, made concrete. $\rho$ is a buffer in value coordinates, and its size is chosen by a quantile over groups in level coordinates. The design has a known limitation: it needs on the order of $1/\delta$ groups. REI-Bench's nine difficulty levels are too few for a guarantee, so we use them only as a leave-one-level-out stress test.

Baselines

The naive reference is a robot that never asks. It shows how much risk is at stake and sets the floor for any question cost.

The strongest existing detection method is CLUE's attention classifier, run with its own decision threshold. CLUE reports 0.846 F1 on its test set and 0.765 on real-world out-of-distribution data. For a fair comparison we retrain it on our calibration split with the same backbone and the same number of tuning trials. We then report its single operating point in risk-versus-questions space. We beat it if, at the risk level CLUE actually achieves on held-out groups, our certified frontier asks no more questions. We also beat it if, at matched question cost, CLUE's realized risk on shifted groups exceeds our certified bound while ours holds.

The strongest calibrated method is KNOWNO-style split conformal prediction. KNOWNO reports 10–24% fewer human interventions than uncertainty-aware baselines at a fixed success target; AmbiK reports it below 20% set size correctness on that benchmark. We run it twice on identical splits: once with its native token-probability scores and once with our attention scores. This shows whether a gap comes from calibration or from the score.

On AmbiRefer3D the reference is D-LISA with the authors' question selection, reported at 41.57% accuracy with 3.39 average rounds. We beat it if some certified point on our frontier reaches equal or better accuracy with fewer rounds. For dialogue-token detection we include Ask-to-Clarify's ambiguity token; no reported result found on Lacuna for it on InViG or AmbiRefer3D.

Three ablations isolate the key ingredients. Pooled CREME with $\rho=0$ tests whether the group buffer matters. A buffer set by a Wasserstein concentration bound tests whether data-driven sizing beats theory-driven sizing. Token-probability scores substituted into the full method test the AmbiK claim that the score is the bottleneck.

Why not just…?

Why not just sweep the threshold on a validation set and plot the empirical trade-off curve? This is the most serious competitor, and it may be nearly as good when validation data are plentiful and exchangeable with deployment. CREME's argument is that an uncorrected sweep underestimates risk and that choosing a point after viewing the curve introduces selection bias. Neither effect has been measured in robotics. We run the sweep as a baseline. If its realized risk on held-out groups stays below its own curve as often as ours does, we will report that certification adds little in-distribution. Our claim would then rest entirely on the shifted-group results.

Why not just use pooled conformal risk control at one target, as KNOWNO does? A single target hides the cost of questions. KNOWNO's guarantee also assumes exchangeability, which REI-Bench's 36.9% drop on implicit references calls into question. The $\rho=0$ ablation is this baseline extended to a full frontier. Its violation rate on held-out scenes and on REI-Bench levels, compared with ours, settles whether the buffer is needed.

Why not just reweight for covariate shift or retrain with a Wasserstein penalty, as WR-CP does? Importance weights need density ratios over instructions and scenes, which are hard to estimate. The penalty requires backpropagating through the grounding model, which is impossible with a frozen vision-language model. Where we can fine-tune the attention head, we will run WR-CP's regularizer as an upper reference.

Why not just prompt the model to say when it is unsure? AmbiK reports that prompting either always asks or never asks and separates ambiguous from unambiguous pairs under 10% of the time. CLUE's prompting baseline reaches 0.683 F1. We include binary prompting as a single point on our plots.

Proposed Experiments

We use four benchmarks, each probing a different kind of shift. InViG-21K tests 2D interactive grounding; we group by image source and evaluate on CLUE's real-world out-of-distribution split. AmbiRefer3D, with 7,316 scenes, has enough exchangeable groups for the scene-level guarantee to be meaningful. REI-Bench lets us calibrate on explicit, standard-context instructions and test on implicit and noised levels, a controlled language shift. AmbiK tests text-only planning, with groups by category (preferences, common sense, safety); the safety category is where risk matters most. A small tabletop study with several users per session provides the user-level test on hardware.

The metrics are the realized error rate per held-out group, the fraction of groups whose realized curve exceeds the certified frontier (target at most $\delta$), questions per task at matched certified risk, AmbiK's set size correctness, and grounding accuracy at a fixed round budget.

Three checks are decisive. First, on held-out AmbiRefer3D scenes and shifted REI-Bench levels, pooled CREME and the empirical sweep must violate their curves noticeably more often than $\delta$, while the buffered frontier stays at or below $\delta$. If pooled methods already hold, the shift argument fails. Second, $\rho$ must be much smaller than the Wasserstein concentration buffer; if the two are similar, data-driven sizing buys nothing. Third, with token-probability scores in place of attention scores, the buffered frontier should become vacuous, meaning it never permits acting below moderate risk. That would confirm AmbiK's finding inside a calibrated system and show that grounded scores are what make the guarantee usable.