Asking for Help Where the Safety Boundary Is Unclear

A robot learning by trial and error has two ways to avoid a crash it cannot yet predict. It can stay conservative and never get near danger, or it can ask a teacher. Each has a well-known failure. Conservative safe learners rarely see near-misses, so their estimate of where danger begins stays wrong; this is the "safety paradox" described in Breaking Safety Paradox with Feasible Dual Policy Iteration. Robots that ask for help usually decide when to ask from generic confusion or from the chance of failing the task, so they spend the teacher's attention in the wrong places and can miss a fast drift toward a hazard. This note proposes one specific way to decide when to ask. The robot asks exactly when its own estimate of future constraint violation is genuinely split: the optimistic reading says the next action fits within the remaining safety budget, and the pessimistic reading says it does not. Every answer the teacher gives is then used twice. It keeps the robot safe at that moment, and it supplies a labelled example at the safety boundary, which is the kind of data a cautious learner is otherwise starved of.

The bet is that a fixed budget of questions, spent only in this ambiguous zone, buys three things at once. First, fewer violations during training. Second, a better-calibrated safety estimate, so the final policy is less timid. Third, fewer environment steps to reach good performance. The comparison is against both autonomous safe reinforcement learning and existing help-seeking imitation methods, all given the same question budget.

Core Claim

We use the constrained Markov decision process: the agent maximizes task reward while keeping expected cumulative constraint cost (collisions, hazard entries) below a budget. We add a communication action whose price is counted against a second, explicit query budget, instead of being folded into the reward with a hand-tuned penalty. The safety critic is a distributional critic, meaning it predicts quantiles of future cost rather than a single number, as in the truncated quantile critics of COX-Q. Let $\hat C_{lo}(s,a)$ and $\hat C_{hi}(s,a)$ be a low and a high quantile of predicted cost-to-go for proposed action $a$ in state $s$, and let $b_t$ be the safety budget remaining at step $t$. The gate fires when

$$\hat C_{lo}(s,a) \le b_t < \hat C_{hi}(s,a) \quad \text{and} \quad \Pr_{\text{critic}}\left[C(s,a) > b_t\right] \ge \tau_t ,$$

where the second condition asks that the critic place at least probability $\tau_t$ on the action overrunning the budget. The threshold $\tau_t$ is raised or lowered by dual ascent, so that average query spend tracks a user-set budget $B$. This replaces a brittle fixed penalty with a target the user can state directly.

When both quantiles say "safe", the agent acts. When both say "unsafe", the agent already knows the answer: it changes its own action through the constrained actor and does not spend a question. Questions go only where the agent does not know.

This is the main difference from ThriftyDAgger. ThriftyDAgger gates on the variance across an ensemble of policies and on a task-completion value function, with budget-matched quantile thresholds. It therefore spends questions in states that are risky but well understood, and its risk signal measures failure to reach the goal, not proximity to a constraint. It also differs from HACO and RLIF. In both, a human who watches continuously decides when to intervene, and interventions shape a proxy value (HACO) or serve as a $-1$ reward (RLIF). Neither lets the agent choose when to ask, and neither turns the answers into constraint-boundary supervision.

Research Program

The system has four parts.

The learner is an off-policy constrained actor-critic in the style of COX-Q, with reward and cost quantile critics.

The query gate is the one defined above, with its dual-adapted threshold.

The teacher answers in one of two ways, and these have different prices. A cheap verdict query asks "would this action lead to a violation within the next few steps?" and returns yes or no. An expensive takeover query hands control to the teacher for a short corrective segment. The agent chooses between them by expected value per unit price. A verdict suffices when only the safety critic needs resolving. A takeover is warranted when the predicted pessimistic cost is high enough that acting on a wrong "yes" would itself be costly. The two-price menu is the cost-aware communication contribution. It draws on the help-seeking line of work, such as Ask4Help and A Framework for Learning to Request Rich and Contextually Useful Information from Humans, which learns when and what to ask in embodied tasks but without any safety constraint.

The two-use data path works as follows. Verdict answers become direct supervised targets for the cost critic at the queried state and action. These are labelled examples on both sides of the boundary, the very data the safety paradox withholds. Takeover segments go into a demonstration buffer that is mixed into off-policy updates, RLPD-style as in RLIF. This corrects the student's drift away from expert-visited states on exactly the recovery states the student reaches, the classic DAgger effect from Ross et al..

The analysis part asks a simple, checkable question: does the straddle region shrink over training? If verdict labels concentrate where the critic is wrong, the gap between the quantiles near hazards should narrow faster than under random or task-risk queries. The query rate should then fall on its own, without the budget forcing it down. We also report how often a violation occurred in a state where the gate did not fire, which measures the gate's misses.

Baselines

The naive reference point is the same constrained actor-critic with no communication channel, run to the same step count. It shows what the question budget buys at all. For context, the file reports that PPO-Lag achieved "a 98% reduction in safety violations compared to standard PPO, with only a 45% drop in total reward" on Safety-Gymnasium velocity tasks (Safety Gymnasium). We include PPO-Lag from the SafePO library as the on-policy anchor.

The strongest autonomous competitor is COX-Q itself. It addresses training-time violations and sample efficiency together, so it isolates what teacher access adds. The file reports only qualitative results for it: lowest collision rates on SMARTS, and stable training cost on Safe Velocity versus ORAC's spikes. So: no reported numeric result found on Lacuna for COX-Q. We will rerun it with the authors' settings.

The strongest help-seeking competitor is ThriftyDAgger. It reports a 73% autonomous success rate on Robosuite peg insertion, against 48% for LazyDAgger and 24% for SafeDAgger. We run it with the same simulated teacher and the same total query price, converted at our verdict and takeover rates. We also run HACO on MetaDrive. HACO reports high performance in 30,000 steps with about 30 training collisions and a 0.83 success rate on unseen maps. We note plainly that these figures come from real humans who monitored continuously. Our rerun uses a scripted teacher, and we charge HACO for every step of monitoring, not only for takeovers.

Three ablations isolate the key ingredient. The first gates on the mean cost exceeding the budget instead of on the quantiles straddling it. The second keeps the gate but discards verdict labels, so questions only prevent violations and never train the critic. The third queries at random under the same budget. Fairness rules are the same across all runs: identical environment-step budgets, identical query-price budgets, the same teacher, five or more seeds, and equal hyperparameter-search trials per method.

Beating the baselines means two things. First, at a matched query price, fewer cumulative training violations than ThriftyDAgger and HACO, with final return no lower. Second, higher final return than COX-Q at matched violation level, reached in fewer steps.

Why not just…?

Why not just swap a cost critic into ThriftyDAgger's risk gate? This is the most serious threat, and it is precisely our first ablation, mean-cost gating with ThriftyDAgger-style quantile thresholds. If it matches us, the novelty collapses to "use a cost critic", and we will say so. Our prediction is that it overspends in states that are clearly dangerous and already handled by the constrained actor. It also gets no boundary labels, so its critic stays as miscalibrated as an autonomous one. Comparing the critic's calibration error near hazards between the two settles this.

Why not just penalize questions and crashes in one scalar reward? The file cites direct evidence that this is brittle on both axes. Quanyi Li et al. describe agents trading crashes for return or freezing. Enabling Intelligent Interactions between an Agent and an LLM reports that query penalties swing agents between over-asking and silence. We include a sweep over penalty weights as a supplementary baseline. The claim to test is that no single weight matches the dual-adapted gate across all three budget levels.

Why not just use a safe fallback controller, as in SOOPER? Safe Exploration via Policy Priors achieves zero training violations, but only given a valid safe prior. A fallback also teaches the critic nothing about where the boundary lies. Where a prior exists, SOOPER is a fair competitor, and we report it on the Safety-Gymnasium tasks where a scripted conservative prior can be written. On MetaDrive's unseen maps we argue that a prior is unavailable, which is where a teacher is needed.

Why not just let a teacher watch continuously, as in HACO and RLIF? That works, but it hides the main cost: attention. Charging monitoring time per step makes the comparison honest. If HACO still wins under that accounting, agent-initiated asking is not worth it in driving.

Proposed Experiments

The main testbed is Safety-Gymnasium navigation (PointGoal, CarButton, PointPush), set up with sparse, small hazards so that violations are rare and the safety paradox bites. Here a scripted lookahead oracle can answer both query types exactly.

MetaDrive tests transfer to unseen road networks against HACO, with a trained expert policy standing in for the human. Robosuite peg insertion tests whether a constraint-defined gate helps or hurts relative to ThriftyDAgger on a task built around task risk.

The metrics are:

  • cumulative training violations;
  • final normalized return and cost;
  • environment steps to reach 90% of final return;
  • total query price spent, with verdicts and takeovers reported separately;
  • the gate miss rate;
  • cost-critic calibration within a fixed distance of hazards.

Three checks decide the outcome. First, the full method must beat the label-discarding ablation on both training violations and steps-to-performance; otherwise the two-use claim fails. Second, it must beat mean-cost gating at equal price; otherwise the straddle gate is unnecessary. Third, sweeping the query budget $B$ must trace a curve on which the method dominates COX-Q at every nonzero budget. A curve showing that a small number of verdicts recovers most of the gain would be the strongest single result.