Robots that learn by trial and error need someone to put the world back after each attempt: re-placing a knocked-over mug, righting a fallen quadruped, retrieving a peg that rolled off the table. "Reset-free" reinforcement learning (RL) tries to remove this person by training a second policy, the backward policy, whose job is to undo what the task policy did. These methods quietly assume that every mistake can be undone. Real hardware breaks that assumption. Some mistakes, such as a peg off the table or a robot on its back, put the system in a state no learned controller can leave. When that happens, a human intervenes anyway, and current methods treat the intervention as an embarrassing accident instead of a resource to plan around.
We propose to make human interventions the explicit constraint. The agent gets a budget of manual resets over its lifetime and learns a calibrated estimate of the probability that it can still get home from where it is. A Lagrange multiplier tied to the observed intervention rate decides how much irreversibility risk the task policy may accept at each point in training. Each actual intervention becomes a labeled example of an irreversible state, which is the data that strictly safe methods never collect. The question is whether a learner starting from scratch, with no prior knowledge, can stay under a fixed human-effort budget throughout training, not only at convergence, while reaching task performance that fixed-threshold safety rules cannot match at the same effort.
Core Claim
Existing reset-free methods either ignore irreversible "sink" states or guard against them with a fixed rule. Leave no Trace trains a forward task policy together with a reset policy. It aborts the task whenever a pessimistic ensemble of reset Q-values drops below a hand-set threshold, so safety means reversibility. That paper reports manual resets as an outcome, 2.5x fewer in Pusher, but never constrains them. Intelligent Switching for Reset-Free RL (RISC) decides when to switch between task and reset modes using a "success critic" that estimates the agent's competence. It has no safety notion, and the file notes that its switching breaks down once that critic collapses to zero in an irreversible state. Safe Exploration via Policy Priors (SOOPER) keeps a cumulative cost budget during training by falling back to a safe prior policy. However, it requires episodic resets and a pre-built safe prior, and its authors name the single-life setting as open.
Our claim is that the right control variable for the reset bottleneck is the number of human interventions, enforced as a constraint over the whole continuing lifetime. A fixed abort threshold is the wrong control variable for two reasons. The appropriate risk level changes as the reset policy improves, and a fixed threshold cannot spend the human's effort where it buys the most learning. Writing $I_t \in \{0,1\}$ for whether a human had to intervene at step $t$ and $B$ for the intervention budget over a lifetime of $T$ steps, the problem is
$$\max_{\pi_f,\pi_b}\; \mathbb{E}\Big[\sum_{t=0}^{T} r(s_t,a_t)\Big] \quad \text{s.t.} \quad \mathbb{E}\Big[\sum_{t=0}^{T'} I_t\Big] \le B\,\frac{T'}{T} \;\; \text{for all } T' \le T,$$
where $\pi_f$ and $\pi_b$ are the forward (task) and backward (reset) policies. The prefix form of the constraint is the point: the budget must hold during training, not only asymptotically. The file documents the gap this addresses. Lagrangian safe-RL methods incur many violations before their multipliers settle (Safety Gymnasium).
Research Program
The method has three parts.
A return-probability critic. The first part is a critic $R(s)$: the probability that the backward policy reaches a home region within $H$ steps from state $s$. It is trained by temporal-difference learning. Mode switches are handled as non-terminal timeouts, which RISC showed is necessary for consistent values across the forward–backward boundary. States where a human had to intervene are written to the replay buffer as absorbing failures with $R=0$. Unlike Leave No Trace, which needs a privileged distance-to-start reward, $R$ only needs a success indicator for reaching home and the intervention log. Unlike RISC, a zero value here is meaningful: it marks a sink to avoid, not a mode to keep retrying.
Budget-coupled control. The second part ties the forward policy's behavior to the budget. The forward policy optimizes $r - \lambda\,(1 - \underline{R}(s'))$, where $\underline{R}$ is the pessimistic ensemble lower bound on the next state's return probability. A PID-controlled update adjusts $\lambda$ from the gap between the realized intervention count and the prorated budget line $B\,T'/T$, borrowing the PID-Lagrangian idea that Safety Gymnasium reports reduces oscillation. Switching combines RISC's competence signal with the same multiplier. The agent hands control to the backward policy when it is already competent at the current frontier, or when $\underline{R}$ falls below a level that rises with $\lambda$. A small hard floor on $\underline{R}$ stays in place as a last-resort abort.
Weighted intervention costs. The third part replaces the 0/1 indicator $I_t$ with type-dependent costs, separating "stuck but harmless" from "damaging" interventions. This addresses the "all failures cost the same" limitation the file attributes to Leave No Trace. It is an extension tested after the main result.
We also analyze when the method helps. We will measure how $R$'s calibration on sink states scales with the number of interventions spent. We will test the hypothesis that front-loading the budget, by spending interventions early, beats spreading it evenly. That hypothesis is our reading of the safety paradox described in Breaking Safety Paradox with Feasible Dual Policy Iteration: without failure examples, feasibility estimates degrade.
Cold start enters as the setting, not as a separate method. All main runs begin with randomly initialized networks. A secondary condition adds a few demonstrations in the style of MEDAL to measure how much prior knowledge reduces the budget needed for a given performance level.
Baselines
Naive reference. The do-nothing reference is standard forward–backward reset-free RL with fixed-interval switching and no irreversibility handling, plus an oracle-reset episodic agent as the upper bound. The file reports that on Sawyer-Peg and Franka-Kitchen "all autonomous algorithms failed to make significant progress, while the Oracle solved them easily" (Autonomous Reinforcement Learning: Formalism and Benchmarking). In our setting the naive agent should consume interventions with no control over the rate.
Leave No Trace. The most direct competitor is Leave no Trace, because it already uses a reset-value ensemble to avoid irreversible states. Its reported numbers are 2.5x fewer manual resets than standard approaches in Pusher and "nearly zero" in Cliff Cheetah. We will run it with the same ensemble size, network sizes, replay ratio, and environment steps as our method. Its abort threshold gets the same tuning budget we give our PID gains. We will also sweep that threshold to trace its full performance-versus-interventions frontier.
RISC. RISC is the strongest method on the deployment-success side of EARL; it "matched or exceeded the best-known results on 3 out of 4 EARL tasks." The file gives no per-task success numbers for it, so no reported result was found on Lacuna for RISC's exact EARL scores, and we will reproduce them.
MEDAL. For the demonstration condition, MEDAL (A State-Distribution Matching Approach to Non-Episodic Reinforcement Learning) reports 40% success on Sawyer-Peg where prior autonomous methods scored 0%. That is the number to beat when demonstrations are allowed.
What counts as winning. Beating a baseline means one of two things. Either we reach higher deployment success at an equal or smaller realized intervention count, or we reach equal success while staying under the prorated budget line at every point in training, which no fixed-threshold method controls.
Ablations. Three ablations isolate the key ingredients. The first replaces the adaptive multiplier with a fixed one. The second removes intervention-labeled sinks from $R$'s training data. The third treats switches as terminal instead of non-terminal.
Why not just…?
Why not just tune Leave No Trace's abort threshold to the desired reset count? This is the real threat. A well-chosen fixed threshold might trace the same frontier as our adaptive multiplier. Our argument is that the right threshold drifts as the backward policy improves, so any fixed value is too timid late in training or too reckless early. That is an empirical claim. The threshold sweep described above settles it. If some fixed threshold matches our success-versus-interventions curve on most tasks, the budget formulation adds nothing beyond a tuning procedure, and we will report that.
Why not just add a large penalty to the reward whenever a human intervenes? The file cites evidence (Quanyi Li et al.) that scalar penalties either tolerate periodic catastrophes or produce agents that freeze. Penalties also provide no control over the transient rate. We will run RISC with intervention penalties at several scales and report both success and the time course of the realized intervention rate. The test is whether any penalty scale stays under the budget line without collapsing task progress.
Why not just engineer the environment to be reversible, with walls, trays, and scripted resets? Where this is cheap, it should be done. The file's examples include cyber-physical systems that cannot be rebooted (Yangyang Fu et al.) and quadrupeds that flip (URSA), and these show that some sinks cannot be engineered away. Formal reach-avoid filters such as Back to Base do guarantee return, cutting unsafe trajectories from 99.4% to 0.07% on Cartpole. However, they need known dynamics and grid-based solvers that do not scale to a 9-degree-of-freedom arm. We include one low-dimensional reach-avoid task specifically to show where the filter wins and where our method is needed instead.
Why not just start from demonstrations? Demonstrations help, as MEDAL shows. Our demonstration condition measures how much budget they save, so this alternative is quantified rather than dismissed.
Proposed Experiments
Environments. We use the Leave No Trace non-episodic suite (Gridworld with absorbing states, Pusher with table-edge sinks, Cliff Cheetah, Cliff Walker, Peg Insertion) and EARL's Sawyer-Peg, Tabletop-Organization, and Franka-Kitchen. In each EARL task we disable the periodic automatic reset and log a human intervention whenever an object leaves the reachable workspace or no progress is made for a fixed window. The reach-avoid Cartpole provides the comparison with formal filters.
Metrics. We measure deployment success from the standard initial distribution in EARL's deployment setting. We also track cumulative interventions plotted against the prorated budget line, the maximum overshoot of that line, and success at matched intervention counts. Every result is reported over multiple seeds and at budgets $B$ ranging from roughly the naive agent's count down to near zero.
Decisive checks. The first decisive check is the frontier comparison against the Leave No Trace threshold sweep on Pusher and Sawyer-Peg; this alone decides whether the idea stands. The second is budget tracking: whether the realized intervention curve stays under the budget line from the first thousand steps onward, compared with the Lagrangian transients reported in Safety Gymnasium. The third is the sink-label ablation, which tests whether interventions as training data are what keep $R$ calibrated. If removing them leaves performance unchanged, the safety-paradox argument for spending the budget is wrong. A final scheduling experiment compares front-loaded and even budget spending and tells practitioners when to be present at the robot.
References
- Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning
- Intelligent Switching for Reset-Free RL
- Safe Exploration via Policy Priors
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Autonomous Reinforcement Learning: Formalism and Benchmarking
- A State-Distribution Matching Approach to Non-Episodic Reinforcement Learning
- MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning
- How good are learning-based control v.s. model-based control for load shifting? Investigations on a single zone building energy system
- From Tabula Rasa to Emergent Abilities: Discovering Robot Skills via Reset-Free Unsupervised Quality-Diversity
- Back to Base: Towards Hands-Off Learning via Safe Resets with Reach-Avoid Safety Filters