Frozen Bundles: Reset-Free Robot Learning With Few Policy Redeployments

A robot learning on real hardware faces three practical costs that most reinforcement learning (RL) algorithms ignore. Every time a person has to walk over and put the scene back in order, that is a manual reset. Every time new network weights are pushed to the machine, someone has to stop it, check the update, and restart it, which is a policy switch. And every time the robot does something it cannot undo, such as knocking an object out of reach or tipping over, the run is either damaged or over. Research has attacked each cost separately. Reset-free RL trains a second "reset" policy that undoes the task policy's work. Deployment-efficient RL limits learning to a handful of weight updates. Safe exploration keeps the agent away from catastrophic states. Each line assumes the others' problem away. Low-switching methods assume episodes that end and restart automatically. Reset-free methods update the policy after every step. Both inherit a cold-start problem, because a randomly initialized safety check cannot tell a safe action from an irreversible one.

We propose to treat each deployment as a frozen bundle. A bundle contains a task policy, a reset policy, and a pessimistic reversibility check, and none of the three changes during the deployment, so a deployment is never revised mid-run. Inside a deployment, the agent switches freely between task and reset controllers, which costs nothing because no weights change. Weights change only between deployments, and only at moments the reset policy itself creates: when the scene has been returned to its start region. Two consequences follow, and the project tests both. First, the reset policy does a second job. It creates the episode boundaries that low-switching criteria need in order to be stable, which removes the main obstacle to using those criteria in never-ending lifetimes. Second, safety has to be budgeted for the whole frozen deployment, because nothing can be corrected until it ends. The abort threshold therefore has to tighten as deployments get longer, and the first bundle has to be safe before any online data exists, which requires warm-starting all three components from demonstrations.

Core Claim

Our claim is that, in reset-free environments that contain irreversible states, a learner can reach success rates comparable to continuously updated reset-free methods. It would use roughly 5 to 20 weight redeployments instead of hundreds of thousands of updates, and enter fewer irreversible states. This requires three ingredients, each of which the ablations isolate. The redeployment decision is checked only at reset-policy-created boundaries. The abort threshold is calibrated to the planned deployment length. The reversibility check is warm-started pessimistically from demonstrations.

The distinction between switching controllers within a run and switching weights between runs is what makes this tractable. Intelligent Switching for Reset-Free RL (RISC) learns when to hand control from the task policy to the reset policy using a learned estimate of how likely each policy is to succeed. It then keeps learning online, so its weights change continuously. The low-switching literature counts only weight changes. Beyond Information Gain: An Empirical Benchmark for Low-Switching-Cost Reinforcement Learning shows that, for neural networks, switching when internal features drift in cosine similarity beats theory-derived information-gain rules. That paper also reports that checking the criterion only at episode starts was necessary for stability. A reset-free lifetime has no episode starts unless the agent makes them. We put these two ideas together: RISC-style handoffs happen inside a frozen deployment, and feature-drift-style redeployment decisions happen at the boundaries those handoffs produce.

Research Program

The deployed bundle. The task policy, the reset policy, and an ensemble of reset-policy value functions are all frozen during deployment. The ensemble is used as in Leave no Trace, which aborts the task policy whenever the pessimistic value of the reset policy falls below a threshold, treating "the reset policy can still get back" as the definition of safe. Leave no Trace keeps its thresholds fixed and updates continuously, and it depends on an accurate ensemble that does not exist early in training. We keep its abort rule but change the two things that break when updates are rare.

Deployment-length-aware abort threshold. Let $\tau$ be the abort threshold, $N_k$ the number of environment steps planned for deployment $k$, $\hat{\epsilon}_k(\tau)$ the estimated per-step probability that the frozen bundle enters an irreversible state despite the abort rule, and $B_k$ the allowed number of irreversible entries (equivalently, manual resets) for that deployment. We choose the most permissive threshold satisfying

$$N_k \, \hat{\epsilon}_k(\tau) \le B_k,$$

which is a union-bound budget over a deployment during which nothing can be corrected. We estimate $\hat{\epsilon}_k(\tau)$ from the previous deployment's logs. The estimate combines the observed rate of irreversible entries, the rate of near-threshold events, and ensemble disagreement, and it is checked against held-out logged transitions. The rule exposes a trade-off that existing methods never face. Longer deployments mean fewer redeployments but force a stricter threshold, and a stricter threshold means more aborts and less task progress. We will measure this frontier directly.

This formulation also connects to Provable Reset-free Reinforcement Learning by No-Regret Reduction. That paper casts manual resets as violations of a constraint and proves sublinear regret and sublinear resets in linear models. Its dual variable is updated every episode. Ours is effectively updated once per deployment, and its constraint is stated per frozen deployment rather than per episode.

Boundary-checked redeployment. A boundary event occurs when the reset policy brings the system back to the start region. At each boundary event, the learner computes the feature-drift statistic of Beyond Information Gain between the deployed networks and networks trained offline on all data collected so far. It also checks whether the deployment's safety budget is nearly spent. If either fires, the learner redeploys. Otherwise it continues with the same bundle. Between deployments, all three components are retrained offline on the full replay data. For the value functions we use the periodic bootstrapping truncation that Autonomous Reinforcement Learning: Formalism and Benchmarking found necessary for stable temporal-difference learning on very long trajectories, together with RISC's timeout-nonterminal bootstrapping at handoffs.

Pessimistic warm start of all three components. The first bundle has to be safe before any online data exists. Constrained Markov Decision Processes via Backward Value Functions states that its guarantees need a feasible initial policy, and Leave no Trace's aborts are unreliable before its ensemble is trained. We initialize the task policy by behavior cloning on the demonstrations that EARL supplies, following the warm-start recipe of BREMEN. That paper reaches MuJoCo locomotion in 5 to 10 deployments from such an initialization, but with episodic resets and no safety constraint. We initialize the reset policy by offline goal-conditioned training on the same demonstrations, relabeled so that reaching the demonstrations' initial states counts as success. The reversibility ensemble is trained so that states far from demonstration support receive low values. As a result, the first deployment aborts early wherever the data cannot vouch for reversibility. This adapts BREMEN's behavior regularization to the safety check rather than the policy. The demonstrations carry no reward labels for the reset task, so the goal-relabeling route avoids the reward-annotation requirement of Reinforcement Learning from Imperfect Demonstrations.

Baselines

The naive reference point is a reset-free learner with a fixed time-based switch between task and reset policies and continuous online updates. Wherever it enters an irreversible state, we count one manual reset. This reproduces the "naive" and fixed-switching conditions that the EARL paper and RISC already show to be weak. It anchors the manual-reset and success axes, and it reports its number of weight changes honestly, which is on the order of the number of gradient steps.

The strongest competitor on success under reset-free conditions is RISC, run with its published configuration and continuous updates. It is the best method in the file on EARL: it "matched or exceeded the best-known results on 3 out of 4 EARL tasks" and approached oracle-reset performance on Sawyer-Door and Tabletop. No reported per-task success numbers for RISC were found on Lacuna, so we will rerun it under our environment-step budget and seeds. Beating RISC on its own terms is not required. Our success criterion is to land within the 95% confidence interval of RISC's deployment-setting success on Tabletop and Sawyer-Door while using at most 20 redeployments and entering fewer irreversible states. On Sawyer-Peg, where both are expected to struggle, we would need to exceed it.

The strongest competitor on safety in irreversible environments is Leave no Trace. On its own suite, it reports a 2.5× reduction in manual resets on Pusher and near-zero resets on Cliff Cheetah. We rerun it with continuous updates, and also in a frozen-bundle variant in which its fixed threshold is kept across our deployment schedule.

The strongest deployment-efficient competitor is BREMEN given the same demonstrations and the same deployment count. Because it needs episodic resets, every episode boundary it requests is counted as a manual reset. Its published comparison is 5 to 10 deployments against 100,000 to 1,000,000 for SAC in episodic MuJoCo. We expect it to be competitive on deployments and poor on manual resets, and we will report both.

Every method receives the same demonstrations, the same environment-step budget, and the same offline compute per deployment. Each also gets the same hyperparameter search budget of twenty configurations, tuned on a held-out task variant.

Three ablations isolate the ingredients. The first replaces boundary-checked redeployment with a fixed redeployment schedule at matched deployment count. The second uses a fixed abort threshold instead of the length-aware one. The third trains the reversibility ensemble from scratch instead of warm-starting it pessimistically.

Why not just…?

Why not just freeze Leave no Trace or RISC and redeploy on a fixed schedule? This is the most serious threat. If a fixed schedule at matched deployment count does as well as boundary-checked redeployment, the claim shrinks to "frozen bundles plus warm start suffice, and the trigger does not matter." The evidence against it is indirect. Beyond Information Gain found that checking switches at arbitrary times was unstable and that drift-based triggers beat information-gain triggers on Atari, with an average reward-threshold switching improvement of 3.60 versus 2.33. Atari is episodic, however. The first ablation settles the question directly, and we will report whichever result holds.

Why not just learn offline from the demonstrations and never deploy more than once? This would need zero redeployments, using a safe offline method such as Constraint-Adaptive Policy Switching, which picks among several offline-trained policies at test time and satisfies constraints on 34 of 38 benchmark tasks. That method assumes episodic data and gives the reset policy no way to improve. BREMEN reports standard offline methods failing completely with realistically small datasets on FetchReach and HandReach. We include a single-deployment variant of our own pipeline, which is the warm-started bundle alone, as the point at the left edge of the deployment-count curve.

Why not just penalize irreversible states in the reward? Safe Reinforcement Learning via Constrained Markov Decision Processes documents the familiar failure of this approach. A small penalty is accepted in exchange for reward, while a large penalty produces an agent that does nothing. A penalty also cannot prevent a sink entry within a frozen deployment, because the agent learns from it only at the next redeployment. We run a reward-penalty variant at three penalty scales as a further ablation of the abort mechanism.

Why not just learn in a world model's imagination? URSA learns locomotion skills reset-free on a real quadruped using imagined rollouts and a safety critic, but it updates continuously and states that human intervention is still needed when the robot tips over. Imagination reduces real samples. It neither bounds redeployments nor protects the first real data collection, which is exactly where our pessimistic warm start operates. A model-based variant of our between-deployment retraining is compatible with the proposal, but we treat it as future work rather than as a competitor.

Proposed Experiments

We use EARL's Tabletop-Organization, Sawyer-Door, Sawyer-Peg, and Minitaur-Pen tasks with EARL's demonstrations, evaluated under both EARL protocols: success from fixed initial states, and lifetime reward. EARL tasks are largely reversible, so we add sink-state variants. In these, the mug or peg falling off the table, or the Minitaur leaving its pen, ends autonomous operation and counts as a manual reset. We also include Cliff Cheetah, Cliff Walker, and Pusher from the Leave no Trace suite, where irreversibility is built in. Every method reports four numbers: number of weight redeployments, number of manual resets (irreversible entries plus requested episode boundaries), task success or lifetime reward, and environment steps.

Four outcomes are decisive. The first is the Pareto frontier of success against redeployments at fixed manual resets. The proposal succeeds if our curve dominates frozen-bundle Leave no Trace and BREMEN, and touches RISC's continuously updated success level at or below 20 redeployments on Tabletop and Sawyer-Door. The second is the length-versus-conservatism trade-off. We sweep $N_k$ and verify that the length-aware threshold keeps realized irreversible entries per deployment near $B_k$, while the fixed threshold exceeds it as $N_k$ grows. If realized entries do not track the budget, the calibration idea fails. The third is first-deployment safety: irreversible entries during deployment one with and without the pessimistic warm start. The fourth is the fixed-schedule ablation, which tells us whether the boundary trigger is a contribution or a nicety. A negative result on Sawyer-Peg, where the EARL paper reports that all autonomous methods failed while an oracle with automatic resets succeeded, would be informative too. It would locate the remaining gap in exploration rather than in switching or safety.