Spending Expert Time on Safety Boundaries: Information-Directed Data Collection for Imitation Learning

A robot that learns by copying an expert drifts into situations the expert never showed it, and the dangerous ones are the situations that matter most. Two families of fixes exist. The first asks the expert to take over when the learner looks unsure. The second deliberately pushes the expert toward obstacles during demonstrations so that recovery maneuvers get recorded. Neither asks what an intervention is worth. Takeover rules fire on fixed or quantile thresholds of uncertainty. Disturbance rules push toward danger whether or not the learner already knows how to recover there. The result is wasted expert effort in regions the learner has already mastered, and risk taken on for no gain in knowledge.

We propose to treat every data-collection decision as a purchase. At each decision point the system can let the learner act, hand control to the expert, or hand control to the expert while injecting a disturbance that steers toward the safety boundary. It picks whichever option buys the most information per unit of cost about how to act safely in the states the learner will actually visit. Cost here means expert time plus risk of a constraint violation. Options whose predicted violation risk exceeds a hard per-episode budget are never considered. The selection rule comes from information-directed sampling, a bandit exploration rule that picks actions by the ratio of squared regret to information gained. In the bandit setting this rule is known to ask a costly human for help when Thompson sampling never does.

Core Claim

Interactive imitation can reach the same safe-success rate with substantially less expert effort and fewer violations during data collection than either uncertainty-gated takeover or geometry-driven disturbance injection. The condition is that the decision among act, query, and perturb-then-query is made by one rule. That rule weighs cost against information, and it measures information over the learner's future visitation distribution rather than at the current state alone.

Concretely, let $o$ range over the available options at the current state. These are "learner acts", "expert acts", and "expert acts under disturbance of magnitude $k$ toward the nearest failure set." Let $\Delta(o)$ be the option's expected operational cost: expert seconds consumed plus $\lambda$ times the predicted probability of a constraint violation, where $\lambda$ converts risk into expert-time units. Let $g(o)$ be the expected reduction in the learner's uncertainty about the expert's action, averaged over states drawn from the learner's own rollouts. We choose

$$ o^\star = \arg\min_{o \in \mathcal{O}_{\text{safe}}} \frac{\Delta(o)^2}{g(o)}, \qquad \mathcal{O}_{\text{safe}} = \{o : \text{cost-so-far} + \widehat{\text{future cost}}(o) \le B\}, $$

where $B$ is the episode's safety budget. The squared numerator is the standard information-directed sampling exchange rate between regret and bits. The feasibility set borrows the cost-so-far-plus-cost-to-go bound of backward value functions.

Position Relative to Prior Work

The motivating evidence comes from Aligning AI Agents via Information-Directed Sampling. In a bandit where querying a human has a fixed cost, that paper shows Thompson sampling never queries and suffers linear regret, while information-directed sampling queries and achieves sublinear regret. That setting has no states, no dynamics, and no safety constraint. We carry the same argument into closed-loop control. There, "query" splits into a passive takeover and an active perturb-and-takeover, and the regret term must include safety risk.

The ratio itself comes from Learning to Optimize via Information-Directed Sampling. The visitation-averaged information term adapts the marginal information ratio of Contextual Information-Directed Sampling. That work showed that rules which judge information only at the present context explore myopically. The same critique applies directly to current takeover rules, which judge each state in isolation.

The interactive imitation line is the closest competitor. SafeDAgger hands control to the expert when a classifier predicts large action mismatch. LazyDAgger adds hysteresis to reduce context switches, along with isotropic noise on expert actions. ThriftyDAgger gates on ensemble novelty and a learned task-failure value under a user-set intervention budget. All three make a binary takeover decision from thresholds at the current state. None can choose to perturb, none prices information against cost, and ThriftyDAgger's risk signal is task failure rather than a separate safety constraint.

The disturbance-injection line is SAFE-GIL, which pushes the expert toward the failure set using Hamilton-Jacobi reachability, and MPC-guided disturbance injection, which scales the same idea to high dimensions with sampling-based model predictive control. We reuse their disturbance generator as one option in our menu. We differ in deciding when and how hard to perturb based on what the learner still does not know.

Two papers constrain the design. Breaking Safety Paradox with Feasible Dual Policy Iteration shows that very safe data collection starves the learner of boundary information. This is exactly why the disturbance option must exist, and why its use should be rationed by information rather than banned. RAIL shows that a runtime filter around an unconstrained imitation policy yields policies that succeed only by violating constraints. In PointMaze, Diffusion Policy reaches nominal success but only 15% safe success. This argues that safety has to enter the data, not only the deployment wrapper.

Closing the Closed-Loop Distribution Shift in Safe Imitation Learning attacks drift through stability constraints that need known dynamics. We assume no dynamics model beyond the simulator that disturbance generation already requires. Finally, Constrained Best Arm Identification shows that information about feasibility and information about optimality are distinct quantities with distinct sample costs. That result is the reason our information term targets actions near the boundary rather than generic action uncertainty.

Research Program

The learner is a bootstrapped ensemble of policies trained on the aggregated data. The information term $g$ for a candidate option is estimated in three steps. First, roll the option forward in the simulator to obtain the states it would produce. Second, compute the ensemble's disagreement about the expert action at those states after subtracting a per-state noise estimate. A separate head predicts the variance of the expert's own actions, so the disagreement measures reducible uncertainty rather than expert inconsistency. Third, propagate the expected disagreement reduction to states sampled from recent learner rollouts, using a kernel in the ensemble's shared feature space, which amounts to a linearized Gaussian-process update.

That third step is the non-myopic ingredient. A label at a state no future rollout resembles earns little credit. A label at a boundary state the learner keeps approaching earns a lot.

The cost term $\Delta$ uses expert time per option, measured in simulated-expert seconds and counted as context switches in the user study. It adds $\lambda$ times violation probability, which comes from a learned safety-cost critic. A backward cost-so-far estimate enforces the per-episode budget $B$.

We will analyze two regimes. In a tabular or linear-Gaussian toy, we will check whether the bandit sublinear-regret argument survives when information is averaged over visitation. In the continuous benchmarks, we will check empirically whether the rule reallocates disturbances away from mastered boundary regions over training. We expect perturbation frequency to fall in regions where the learner already recovers, and a heat map of where each method spends expert time will show this directly.

Baselines

The naive reference is behavior cloning on a fixed demonstration budget, together with plain DAgger with expert labeling at every step. These bound what costs nothing in decision logic and what costs the most in expert time.

The strongest takeover competitor is ThriftyDAgger. It reports 73% autonomous success on Robosuite peg insertion, against 48% for LazyDAgger and 24% for SafeDAgger. We will run it as specified, and also in a strengthened version in which its risk value is replaced by our safety-cost critic, so it sees the same constraint signal we do.

The strongest disturbance competitor is MPC-guided disturbance injection. On an unseen F1Tenth track it reports 68.15 m mean travel distance, against 48.96 m for random-noise injection (DART) and 25.55 m for behavior cloning. We will run it with the same disturbance generator our method calls, applied on a fixed schedule.

All methods share the same simulated expert, policy architecture, ensemble size, simulator rollouts per decision, and total expert-time budget. Each method's free thresholds and weights get an equal hyperparameter search. A method beats the field if it reaches equal or higher safe success at equal expert time with fewer violations during collection, or reaches the same safe success with clearly less expert time, across seeds.

Three ablations isolate the key ingredients. The first removes the perturb option, giving information-directed takeover only. The second evaluates information at the current state only, removing the visitation averaging. The third replaces the ratio with plain disagreement maximization subject to the budget, which keeps the information signal but drops the cost exchange rate.

Why not just…?

Why not just use ThriftyDAgger with a safety critic in place of its task-failure value? This is the most direct threat, and it is why the strengthened ThriftyDAgger is a baseline rather than a strawman. Its thresholds are still set per state by quantiles. It cannot choose to create boundary data by perturbing, so it inherits the expert blind spot that motivated SAFE-GIL. If it matches us even when the perturb option is available to us, the contribution reduces to the ablation without perturbation. We will report that outcome honestly.

Why not just inject MPC-guided disturbances on every demonstration? Fixed-schedule disturbance spends expert effort and risk on boundary regions the learner may already handle. Its reported advantage over random noise shows that direction matters. It does not show that frequency and location are well chosen. The expert-time-matched comparison and the allocation heat maps test whether rationing disturbances by information gives the same safe success for less effort.

Why not just pick the most uncertain option, without the ratio? If expert time and violation risk were roughly equal across options, uncertainty maximization would make the same choices as the ratio. The ratio matters only when perturbation is much riskier or costlier than passive takeover. The bandit result in Aligning AI Agents via Information-Directed Sampling shows that ignoring cost structure can fail badly, but that result is for a different setting. The third ablation settles it. We will also sweep $\lambda$ and the disturbance cost to locate where the gap opens or vanishes.

Why not just train a strong policy offline and wrap it in a runtime filter? RAIL itself shows that filtered unconstrained policies keep triggering fallback stops. We will include behavior cloning with a filter as a deployment-time reference and compare its rate of fallback triggers and its safe success against our policies.

Proposed Experiments

The primary testbed is Safety-Gymnasium navigation with Point and Car agents in the Goal and Button tasks. The expert is a converged constrained-RL policy from its SafePO library. The safety cost is the benchmark's hazard contact cost. These environments have built-in costs and fast simulation for the rollouts that both disturbance generation and our information estimate require.

The second testbed is Robosuite peg insertion, chosen to anchor against ThriftyDAgger's reported numbers. We will add a keep-out region around the fixture as the constraint. Because of that change, our safe-success figures are not directly comparable to the published 73%. We will therefore rerun all baselines rather than quote them.

The third testbed is PointMaze, where we report RAIL's safe-success metric. RAIL reports 97% safe success with its filter, against 15% for unfiltered Diffusion Policy.

Metrics follow from the claim. The main ones are safe success at deployment, meaning task completion with zero violations, and violations accumulated during data collection. We also measure expert time and context switches, autonomous success without a filter, and safe success as a function of cumulative expert time.

The decisive checks are three. First, at matched expert time, the full method must beat both the strengthened ThriftyDAgger and fixed-schedule MPC disturbance on safe success without exceeding the collection budget. Second, removing visitation averaging must measurably hurt, which would show that non-myopia is doing real work. Third, the ratio must beat uncertainty maximization in the high-disturbance-cost regime.

If the first check fails while the ablations look flat, the conclusion is that existing gating plus a safety critic suffices. That is a useful negative result for the field.