Self-DAgger: Recovering From Drift With Best-of-k Safe Rollouts

A robot policy trained by copying a few expert demonstrations works until it makes a small mistake. That mistake carries it into states the expert never visited, and from there errors pile up. The standard fix, DAgger, asks a human expert to label those drifted states. The alternative, reinforcement-learning fine-tuning, needs a hand-written dense reward, which is fragile to design. We propose a third route that needs neither. From each drifted state the policy actually reaches, the simulator branches k attempts by the policy itself. An attempt counts as good only if it finishes the task and stays within a safety budget. The policy is then trained to raise the chance that at least one of the k attempts is good. In effect, the policy's own lucky attempts replace the expert in DAgger.

The choice of a "best-of-k" objective, rather than plain average success, is the core of the proposal. Average success gives almost no learning signal at hard drifted states where the policy succeeds 2% of the time. Best-of-k optimization is a form of risk-seeking policy gradient, meaning it rewards rare excellent outcomes instead of typical ones. It places its largest weight on exactly those hard states and nearly zero weight on states that are already solved. The safety conjunction keeps the risk-seeking pressure from favoring trajectories that succeed by breaking constraints.

Core Claim

Let $d_{\text{BC}}$ be the distribution of states visited when the behavior-cloned policy runs closed-loop. Let $p_\theta(s)$ be the probability that policy $\pi_\theta$, started at state $s$, completes the task with cumulative safety cost at most a budget $b$. We maximize

$$J_k(\theta) = \mathbb{E}_{s \sim d_{\text{BC}}}\big[\,1 - (1 - p_\theta(s))^k\,\big] - \beta\, \mathbb{E}_{s}\,\mathrm{KL}\big(\pi_\theta(\cdot|s)\,\|\,\pi_{\text{BC}}(\cdot|s)\big),$$

where $k$ is the number of branched attempts and $\beta$ weights a penalty keeping the policy near the cloned one. The gradient of the first term with respect to $p_\theta(s)$ is $k(1-p_\theta(s))^{k-1}$. A state with $p=0.02$ and $k=16$ therefore receives about $11\times$ more weight per unit of improvement than under average-success training. A state with $p=0.8$ receives essentially none. We claim that this per-state reweighting is what turns sparse, unshaped success signals into recovery behavior. We also claim that it beats both success-filtered self-imitation and fine-tuning on an engineered reward on held-out drifted states, at matched compute and with comparable or lower constraint violation.

Relation to Prior Work

The method draws directly on two risk-seeking papers. Pass@K Policy Optimization shows that optimizing average single-attempt success biases toward conservative policies, and it supplies unbiased estimators for best-of-k gradients from $n \ge k$ samples. We reuse those estimators, but we apply them per start state in continuous control rather than per prompt in language reasoning. Deep symbolic regression updates only on the top fraction of sampled candidates and reaches 83.6% recovery on the Nguyen suite. It does so unconstrained, which is acceptable for formula search and unacceptable for robots. Risk-sensitive RL for LLM exploration and epistemic-risk-seeking optimization motivate the claim that risk-seeking objectives escape regions where average-case gradients vanish. Neither paper has a safety budget or a demonstration prior.

On the safety side, the closest work keeps a hand-written reward. Boundary-Seeking Policy Gradient is risk-seeking with respect to the cost budget: it pushes conservative policies toward the constraint boundary. Feasible Dual Policy Iteration trains a second policy that deliberately seeks violations so that feasibility estimates stay informed. Our risk-seeking acts on task success instead, and the constraint enters as a hard per-trajectory filter, so there is no multiplier to oscillate. FDPI's "safety paradox" constrains our design. If safe successes are too rare, the estimator starves, and we measure how often all $k$ branches fail.

On imitation, three methods correct drift in different ways. DAgger labels drifted states with an expert, and ThriftyDAgger rations those labels by gating on novelty and risk. We keep DAgger's choice of which states to train on but remove the expert. SAILOR also removes the expert. It searches a learned world model with a learned discriminator reward and distills the result via expert iteration. We branch in the true simulator with a binary success-and-safe signal, which avoids discriminator exploitation but requires resettable simulation. SAILOR's distillation step treats all improved trajectories alike, and that uniform treatment is precisely the ingredient we test against.

Research Program

The pipeline has four stages:

  1. Train a diffusion or Gaussian behavior-cloning policy on 20–50 demonstrations.
  2. Roll it out to collect drifted states from $d_{\text{BC}}$.
  3. Branch $n=32$ attempts from each state, using simulator state resets.
  4. Compute best-of-k gradient weights with the unbiased estimator and update under the KL anchor.

Repeat stages 2–4 for several rounds so that $d_{\text{BC}}$ tracks the current policy.

The analysis has three parts. First, a per-state breakdown bins held-out drifted states by initial $p(s)$ and reports gains in each bin. Second, a sweep over $k \in \{1, 4, 16, 64\}$ tests the prediction that larger $k$ moves gains toward low-$p$ bins. At $k=1$ the objective reduces to REINFORCE on the binary signal. Third, we count violations during training, because branching from risky states is itself exploration.

We state the limitations plainly. The method needs state resets and is therefore sim-only. The binary signal requires a success detector and a cost function. The cost function is specified, not tuned, but it is not free.

Baselines

The naive reference is the behavior-cloned policy with no further training. A second naive reference adds non-interactive noise injection during demonstration collection, as in Action Chunking and Data Augmentation, with action chunking on manipulation tasks. On locomotion this was reported to match DAgger, with no exact percentage found on Lacuna for RoboMimic Tool Hang.

The key ablation is success-filtered self-imitation. It uses identical branched rollouts but adds every safe success to the behavior-cloning dataset with uniform weight, which is the expert-iteration step used by SAILOR. It shares our data, compute, and the KL anchor, and differs only in per-state weighting. Beating it means higher success on low-$p$ held-out states with no loss on high-$p$ states.

A second ablation drops the safety conjunction, giving unconstrained best-of-k. It isolates whether the filter is what keeps violations down.

For engineered-reward fine-tuning, the strongest competitors are PPO-Lagrangian and FOCOPS. Both are initialized from the same behavior-cloning policy, use the benchmark's dense reward, and receive equal environment steps and an equal hyperparameter-search budget. Safety Gymnasium reports that PPO-Lag reduces safety violations by 98% relative to PPO at a 45% reward drop on velocity tasks. FOCOPS reports reward 1106.1 on Humanoid-Circle versus 671.0 for CPO under a cost limit of 50. Beating this family means matching their success at the same cost budget without using their reward.

BSPG is included on SafetyPointGoal1. It reports higher reward than CRPO and ESPO, but no reported number was found on Lacuna for its value.

Among imitation methods, SAILOR is the strongest expert-free competitor. It is reported to match a Diffusion Policy trained on roughly 200 demos using only roughly 20, but no absolute success rate is given on Lacuna. We run it on the same demos with matched wall-clock compute. ThriftyDAgger serves as an expert-in-the-loop upper reference, using a scripted oracle as the expert. It reports 73% autonomous success on simulated peg insertion versus 24% for SafeDAgger and 48% for LazyDAgger. Approaching it without any expert labels would be the headline result.

Why not just…?

Why not just keep the successful safe rollouts and re-clone on them? This is the most serious threat, because it uses the same data and is simpler. Our argument is that uniform weighting spends most of its updates on states that are already easy, since those produce most of the successes. The per-state breakdown settles the question directly. If filtered self-imitation matches us in the low-$p$ bins, the best-of-k weighting adds nothing and the paper's claim fails.

Why not just fine-tune with PPO-Lagrangian on the benchmark's reward? On benchmarks that ship a tuned dense reward, this may well win on return. The cost of designing such rewards elsewhere is documented in quadruped locomotion work, where velocity, posture, and energy terms conflict, and Lagrangian methods are known to oscillate between reckless and useless behavior. We report this comparison honestly. The claim is parity in success at equal cost without reward design, not dominance.

Why not just fold safety into the reward as a fixed penalty? The material reports that fixed penalties trade catastrophic acceptance for paralysis depending on the weight. Our conjunction has no weight to tune. We include a penalty variant across three weights to show the sensitivity.

Why not just collect more demonstrations? SAILOR reports that ten times more demos did not close the gap with search-based recovery. We also train behavior cloning on $5\times$ the demos as a reference point.

Proposed Experiments

The main benchmarks are Safety-Gymnasium navigation (PointGoal1, CarGoal1, and PointButton1) and velocity-limited Hopper and HalfCheetah. On these, demos come from a trained PPO-Lag expert and cost is the native benchmark cost. We also use RoboMimic Can, Square, and Tool Hang, where cost is the count of contact-force threshold exceedances.

Metrics are:

  • success-and-safe rate from held-out drifted states;
  • full-episode success;
  • episodic cost relative to the budget;
  • violations during training per environment step;
  • the fraction of states where all $k$ branches fail.

Three checks decide the outcome. The first is whether best-of-k beats filtered self-imitation in the lowest two $p$ bins at matched rollouts. The second is whether gains shift toward low-$p$ states as $k$ grows. The third is whether removing the conjunction raises episodic cost while leaving success roughly unchanged, which would show that risk-seeking without the filter finds unsafe shortcuts. If the first check fails, we report the method as equivalent to expert iteration and stop.