Budgeted Candidate Selection for Safe Self-Improvement of Frozen Imitation Policies

Robots trained by imitation, meaning supervised copying of human demonstrations, can only be as good as the demonstrations they copy. A recent approach gets past this limit without retraining the large imitation model. It keeps the model frozen, samples several candidate action chunks (short sequences of motor commands) from it, and lets a smaller learned critic pick among them. The critic learns from the robot's own successes and failures. The catch is that the failures are real. They happen on hardware, and nothing limits how bad they get. This note proposes a way to let the robot improve while spending a fixed, explicitly accounted safety budget, and a way to spend part of that budget on learning where danger actually lies.

The central observation is simple. Once the frozen policy has proposed $N$ candidates, a continuous control problem has become a choice among $N$ discrete options. Constrained decision-making over a finite action set can be solved exactly as a tiny linear program, a fact that continuous-action safe reinforcement learning cannot exploit. We combine three pieces: that exact per-step solve, the observed cost incurred so far in the episode, and a learned estimate of the cost still to come after each candidate. Together they give a selector that trades reward against a hard episode budget at every step, during learning rather than only after convergence.

Core Claim

The frozen-prior, sample-and-score approach should be run as a constrained problem, solved exactly over the sampled candidates. Doing so is enough to make self-improvement past the demonstration ceiling respect a cost budget throughout deployment, at a smaller reward penalty than hard filtering or a single global penalty weight. The approach has a known failure mode, which we call the safety paradox. A cost critic that only ever sees safe behavior loses track of where the danger boundary is. That failure can be controlled without deliberately unsafe exploration: we reserve an explicit slice of the budget and spend it only on candidates the cost critic is most uncertain about.

Concretely, at state $s$ with candidates $a_1,\dots,a_N$ drawn from the frozen policy, the selector solves

$$\max_{w \in \Delta_N} \sum_{i=1}^N w_i\, Q_r(s,a_i) \quad \text{s.t.} \quad \sum_{i=1}^N w_i\, \bar Q_c(s,a_i) \le B - C_t .$$

Here $w$ is a probability vector over the candidates ($\Delta_N$ is the set of such vectors) and $Q_r$ is the learned reward critic. $\bar Q_c$ is an ensemble upper bound (mean plus $\kappa$ standard deviations) on the cost-to-go after executing chunk $a_i$ and continuing to act. $B$ is the episode cost budget, and $C_t$ is the cost actually observed so far in the episode. Both critics are terminal value estimates in the planning sense: each summarizes everything that happens after a short committed lookahead of one chunk. If the program is infeasible, the selector executes the candidate with the smallest $\bar Q_c$.

Relation to Prior Work

The mechanism we build on is Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning. That work keeps a large vision-language-action policy frozen, trains a compact off-policy critic with categorical value bins, and executes a critic-weighted average of sampled chunks. It raised LIBERO-10 success from 93% to 99%, and on real robots it raised stack-cups from 40% to 90% while filtered supervised fine-tuning stalled at 55%. Q-Planning has no cost model. It learns from failures after they happen, and those failures are unbounded. We keep its proposal-and-score structure and change the selection rule and the exploration rule.

The exact finite-action solve comes from Constrained Markov Decision Processes via Backward Value Functions. That paper turns a trajectory-level budget into a per-state constraint, pairing a learned cost-so-far with a learned cost-to-come, and solves a linear program when actions are discrete. For continuous actions it falls back to a Taylor-approximation projection. It reports a suboptimality gap caused by the conservative upper bound. We differ in two ways. The frozen prior's samples give us discreteness without discretizing the robot's action space. And during deployment the cost-so-far is observed rather than estimated, so one of the two estimated terms disappears. This constrains our design: we expect the remaining conservatism to come entirely from $\bar Q_c$, and we measure it.

The safety paradox motivates the probing component. Breaking Safety Paradox with Feasible Dual Policy Iteration shows that feasibility critics degrade once a safe policy stops visiting violations. It fixes this with a dual policy that deliberately probes constraint boundaries, which is only acceptable in simulation. Our frozen prior makes the problem worse, because the candidate support rarely includes dangerous behavior. We replace deliberate violation with budgeted uncertainty-seeking inside the upper bound.

Safe Exploration via Policy Priors (SOOPER) is the closest safe-exploration relative. It also starts from a safe but suboptimal prior and falls back to it under uncertainty, and it reported zero collisions on a real racecar with a policy "twice as good" as the prior. SOOPER learns a new policy using deep model rollouts and a reset assumption. We never train an actor, and we need no world model. World-model methods such as SafeDreamer and USB-RL plan in learned latent dynamics from scratch, and their own summaries note that they can hallucinate safety in states they have not visited. We rule out world models deliberately, because model hallucination is exactly the risk we want to avoid on hardware.

Discrete-action concerns enter as a constraint rather than the headline. The linear program's solution typically mixes at most two candidates. Switching between different candidates from one chunk to the next can produce the jerky motion documented for discrete controllers. Neural Implicit Action Fields shows that smooth continuous action functions matter for contact-rich tasks, so we measure jerk and chunk-boundary discontinuity and treat smoothness as a reported cost of the method.

Research Program

The system has four parts. The first is the frozen imitation policy, used purely as a proposal distribution. The second is a reward critic trained exactly as in Q-Planning. The third is a cost critic, an ensemble of five heads with the same categorical-bin training, fit to the per-step cost signal on all deployment data and on any demonstrations. The fourth is the per-step selector above, which also tracks the observed cost so far.

The probing rule splits the budget into a task share $B_{\text{task}}$ and a probe share $B_{\text{probe}}$. The main program uses only $B_{\text{task}}$. While probe budget remains, the selector may instead pick the candidate with the largest ensemble disagreement on $Q_c$, subject to that candidate's upper bound fitting within the remaining total budget. Probe spending is recorded exactly.

The analysis has three parts. The first asks how much of the gap to an oracle constrained selector comes from critic error versus the finite candidate set. We answer it by replacing $\bar Q_c$ with ground-truth cost-to-go computed from simulator resets. The second tracks cost-critic calibration over training, with and without probing, to see whether the safety paradox appears. We use expected calibration error on held-out near-hazard states. The third runs a support study that deliberately removes safe candidates, to find where the frozen-prior bound makes the problem infeasible.

We state the guarantee honestly. Budget respect holds only to the extent that $\bar Q_c$ really is an upper bound. The method is therefore a calibrated-safety claim, not a formal one, and the experiments measure how often the upper bound is violated.

Baselines

The naive reference point is the frozen imitation policy executed directly, with no critic. It fixes the demonstration ceiling and its natural cost level. It uses the same demonstrations and the same number of evaluation episodes as every other method.

The strongest competitor for exceeding the ceiling is unconstrained Q-Planning as described in Beyond Imitation. The reported numbers are 93% to 99% on LIBERO-10 and 83.8% to 91.4% on RoboTwin. We run it with the same frozen policy, the same $N$, the same reward-critic architecture and the same number of self-improvement iterations. Our method beats it only if cumulative deployment cost is significantly lower at matched final success. Matching Q-Planning's success is the goal; exceeding it is not required.

The strongest safe-RL competitor on MetaDrive is SafeDreamer. It reports a 100% arrival rate with average episode cost 2.2, against 14.1 for PPO with reward shaping. We report it at its published setting and also rerun it under our environment-step budget. Our method counts as competitive if it approaches that cost level with far fewer environment steps, because it starts from demonstrations. On Safety-Gymnasium, USB-RL and FDPI report only qualitative near-zero cost, and we found no reported numeric table for them on Lacuna. We therefore rerun PPO-Lagrangian and SAC-FDPI ourselves, with equal tuning budgets of 20 configurations each.

Three ablations isolate the key ingredients, and all share critics and data with the full method. Hard filtering keeps only candidates whose $\bar Q_c$ fits the remaining budget and then maximizes $Q_r$. A global-multiplier variant scores candidates by $Q_r - \lambda Q_c$ with $\lambda$ adapted by dual ascent. A no-probe variant sets $B_{\text{probe}}=0$.

Why not just…?

Why not just add a cost critic and filter out unsafe candidates? This is the most serious threat. On tasks where most candidates are clearly safe or clearly unsafe, filtering and the linear program give nearly the same answer. The program should differ from filtering in two regimes: when the remaining budget is tight, and when the best-reward candidate sits just over the limit, because there a mixed choice captures reward that filtering discards. The hard-filter ablation measures this directly. If it matches the full method on reward at equal cost across both benchmarks, we will report the exact solve as unnecessary and reduce the contribution to budget accounting plus probing.

Why not just use one Lagrange multiplier and score candidates by reward minus weighted cost? The dual of our program is exactly such a score, but with a per-state multiplier set by the remaining budget. A global multiplier learned by dual ascent is known to satisfy constraints only at convergence. The SOOPER and BVF comparisons in the file show Lagrangian baselines violating constraints early in training, and First Order Constrained Optimization in Policy Space documents the instability of such updates. We compare cost curves over the first iterations, where this difference should be largest.

Why not just fine-tune the imitation model on safe successes only? Filtered supervised fine-tuning discards the failure signal. The file reports it stalling at 55% and 30% on the two real tasks where Q-Planning reached 90% and 80%. It also offers no control over cost during collection. We include it anyway, with safety-filtered successes, as a cheap reference point.

Why not just learn a world model and plan safely in imagination? SafeDreamer and USB-RL do this and report low simulated cost. But they train from scratch, and their summaries in the file note that they can hallucinate safety in unvisited states, which is the failure that matters on hardware. Our comparison under a matched environment-step budget tests whether demonstration priors plus model-free selection reach low cost sooner.

Proposed Experiments

On MetaDrive and on four Safety-Gymnasium tasks (PointGoal2, PointButton1, PointPush1, CarGoal1), we train behavior-cloning policies on deliberately suboptimal demonstrations collected from early PPO-Lagrangian checkpoints. We then run ten self-improvement iterations. The main metrics are final return or arrival rate, cumulative cost across all deployment episodes, and the fraction of episodes exceeding $B$.

On LIBERO-10 we add a cost signal for contact with non-target objects, using the frozen policy Q-Planning used where available. We report success, cumulative cost and jerk.

The decisive checks are three. First, whether the full method keeps the per-episode over-budget rate near zero while reaching within a few points of unconstrained Q-Planning's final success. Second, whether, without probing, cost-critic calibration on near-hazard states degrades over iterations and late-stage cost spikes appear, as FDPI predicts, and whether the probe budget removes those spikes without increasing total cost. Third, whether the linear program beats hard filtering specifically in tight-budget settings. A negative result on the third check is informative and will be reported as such.