Choosing Where to Restart: Safe Self-Chosen Start States for Reset-Free Learning in Noisy, Irreversible Environments

A robot learning on real hardware usually has no simulator "teleport" to put it back at the start of each trial. A person has to put it back, and that human effort caps how much the robot can practice. Reset-free reinforcement learning removes the person by training a second "backward" policy that drives the robot back to its starting configuration. Simulation studies also show that policies trained from one fixed start memorize a narrow path and break when perturbed, while policies trained from many randomized starts learn to recover. These two facts point in opposite directions on hardware. Randomized starts help, but a physical robot cannot be teleported to a random start. It has to drive there, and on the way it can slip, tip, or drop something into a state it can never undo.

We propose that the backward phase of reset-free learning should do more than return home. It should send the robot to a start state that the robot chooses itself, picked to be informative (a place where the task policy is still weak) and to carry a high probability of safe return. That probability is computed from the robot's real, noisy transitions, not from a predicted average path. The agent then gets the robustness benefit of randomized initialization without oracle resets. It also avoids the irreversible failures that make existing reset-free methods stall, which come from assuming either that every state can be undone or that each action has a single certain outcome.

Core Claim

Existing work splits this problem in ways that leave a specific gap. Autonomous Reinforcement Learning via Subgoal Curricula (VaPRL) already picks intermediate start subgoals using a learned value function, but it picks them for task competence and assumes every state is reversible. Intelligent Switching for Reset-Free RL (RISC) switches between forward and backward controllers based on a learned success probability, with no awareness of safety. Leave no Trace treats safety as reversibility. It aborts forward actions when an ensemble of reset critics predicts that the robot cannot get home, but it assumes one narrow home region and was tested in simulated settings. Safe Stochastic Explorer correctly requires that the whole distribution of next states be safe and returnable. However, it relies on Gaussian processes, which scale cubically with data, and on a fixed safe home zone. FGE learns which initial conditions are feasible, but it teleports to them using oracle resets.

The claim is that one learned quantity can unify these. It is the probability that the backward policy returns the robot to a growing set of verified-recoverable states before it falls into an absorbing failure. This quantity can serve both as a shield on forward actions and as the constraint for choosing the next self-selected start state. Because it is a probability under stochastic dynamics, it can be learned without bias from sampled real transitions. Methods that check safety along a predicted average trajectory systematically overestimate it when noise is present.

Research Program

Let $K$ be the known-recoverable set. It starts as the nominal start region and grows to include any state from which the robot has actually returned to $K$. Let $A$ be the absorbing failure set. In practice the robot learns $A$ from examples, using the moments when a human had to intervene as labels. We train an ensemble of critics $R(s,a)$ for the backward policy $\pi_b$. Each critic estimates the discounted probability of entering $K$ before entering $A$. Each critic is trained with the backup

$$R(s,a) = \mathbb{E}_{s' \sim P(\cdot \mid s,a)}\Big[\mathbf{1}_K(s') + \big(1-\mathbf{1}_K(s')\big)\big(1-\mathbf{1}_A(s')\big)\,\gamma\, R\big(s', \pi_b(s')\big)\Big],$$

where $P$ is the true, noisy transition law, $\mathbf{1}_K$ and $\mathbf{1}_A$ indicate membership in the two sets, and $\gamma<1$ acts as a soft horizon. The expectation is over real sampled next states. Slip and disturbance therefore lower the estimate wherever a tail outcome lands in $A$. This is the situation Roderick et al. describe, in which a single unsafe branch invalidates an action. We take a lower confidence bound over the ensemble, written $\underline{R}$, to cover what the robot does not yet know.

The robot uses $\underline{R}$ in two places. First, as in Leave no Trace, a forward action $a$ is allowed only if $\underline{R}(s,a)\ge 1-\delta$. Otherwise control passes to $\pi_b$. Second, when the forward phase ends, the robot does not simply return home. It chooses a next start $s^\star$ from states it has already visited. The choice favors states where the forward task critics disagree most, which is a cheap signal that the task policy is weak there. The choice is limited to states whose backward return probability stays above $1-\delta$. A goal-conditioned "go-to-start" policy, shielded by the same critic, drives the robot to $s^\star$. Controller switches are bootstrapped rather than treated as terminations, following RISC's timeout-nonterminal fix. Over training, the start distribution spreads outward along the frontier of safely recoverable states. This addresses the safety paradox of Breaking Safety Paradox with Feasible Dual Policy Iteration: a cautious agent keeps receiving data near its boundary instead of shrinking into a small, very safe neighborhood.

The deliverables are the algorithm, an analysis showing that a mean-dynamics feasibility check overestimates return probability by roughly the probability mass of the noise that falls into $A$, and a new benchmark variant. That variant is needed because no environment in the file is at once reset-free, non-ergodic, and stochastic.

Baselines

The naive reference point is episodic training with a human reset after every failure, evaluated from fixed starts. It sets the upper bound on intervention count and shows how fragile single-start training is. The EARL benchmark reports that an oracle-reset policy on Tabletop-Organization lost 37.4% of its performance when tested from random starts, while VaPRL lost only 2%.

The strongest reset-free competitor is RISC. It is the strongest because it already concentrates practice at the edge of the agent's competence, and it matched or beat FBRL, R3L, and MEDAL on 3 of 4 EARL tasks. No reported result found on Lacuna for RISC's per-task numbers, so we will rerun it from its released design. We will also run VaPRL, since its subgoal curriculum is the closest existing mechanism for choosing starts; the file reports that it needs 500x to 1000x fewer resets than episodic RL.

The strongest safety-aware reset-free competitor is Leave no Trace. The file reports a 2.5x reduction in manual resets on Pusher and near-zero manual resets on Cliff Cheetah. We will run it in two forms: as published, and with ensemble critics trained on our noisy transitions.

All methods get the same environment steps, network sizes, and replay setup. Each also gets the same hyperparameter budget, tuned on one held-out noise level. Beating these baselines means either strictly fewer human interventions at equal or better deployment success, or a smaller drop in success from nominal to randomized starts at equal interventions.

Two ablations isolate the key ingredients. The first replaces frontier start selection with uniformly random reachable start goals, with the shield kept. It tests whether choosing starts matters beyond simply having many of them. The second keeps everything but computes return feasibility by rolling out a learned single-outcome dynamics model. It tests whether the stochastic treatment matters. We predict its gap to the full method will grow with the noise level.

Why not just…?

Why not just add a large penalty for failures to RISC or VaPRL, or a Lagrangian cost constraint? A penalty is paid only after the robot has entered an absorbing state, and that event is exactly the manual reset we are trying to avoid. Safety Gymnasium shows that scalar shaping either tolerates violations or freezes the agent. Lagrangian methods such as PPO-Lag cut violations by 98% in velocity tasks, but only at convergence, and with a 45% reward drop. We will include RISC with a PID-Lagrangian cost on entering $A$ and count interventions during early training, which is where we expect it to lose.

Why not just learn a world model and practice from random starts in imagination, as MoReFree does? This is a real threat for smooth dynamics, because imagined starts carry no physical risk. MoReFree, however, failed on contact-rich Sawyer Door because its model mispredicted contacts, and it provides no protection during physical exploration. We will run MoReFree on the same tasks. If it matches us on the low-contact tasks but not the contact-rich ones, that supports the claim that real-transition return probabilities are needed exactly where models are weakest. If it matches us everywhere, our contribution narrows to the shield alone.

Why not just use Leave no Trace and send the backward policy to random targets? This is our first ablation. The test is whether reversibility checks plus random targets already capture the benefit. If they match the full method, the frontier selection is unnecessary and we will report it that way.

Why not train in simulation with domain randomization and transfer? FGE reports over 50% more coverage of hard parameters than domain randomization on Dubins and Hopper, because uniform sampling wastes effort on infeasible regions. Sim-to-real gaps also remain. Our method is complementary to this: a policy pretrained in simulation can still be fine-tuned on hardware using our procedure.

Proposed Experiments

The testbed will modify three EARL tasks. In Tabletop-Organization, the mug can fall off the table edge. In Minitaur-Pen, the quadruped can tip over irrecoverably. In Sawyer-Door, the door can jam past its hinge limit. Each will be run with action slip and random force disturbances at three noise levels. We will add Cliff Cheetah and Pusher from the Leave no Trace suite with the same noise, plus the 2D navigation grid with 65% slip from Safe Stochastic Explorer. On that grid, deterministic baselines failed in 65% of trials, which makes it a clean test of the mean-dynamics ablation.

The metrics are four:

  1. Human interventions, counted as entries into $A$.
  2. EARL deployment success from nominal starts.
  3. EARL deployment success from randomized starts, together with the relative drop.
  4. Continuing-setting cumulative reward.

There are three decisive checks. The first is whether the full method shows the VaPRL-like small robustness drop while also cutting interventions below Leave no Trace on the non-ergodic variants. The second is whether the advantage over the mean-dynamics ablation grows monotonically with noise, which would confirm that assuming deterministic transitions is what breaks existing shields. The third is whether frontier selection beats uniform random start targets on randomized-start success. If the second check fails, the stochastic treatment is unnecessary for these tasks and the paper becomes a study of start selection under reversibility shields. If the third fails, the contribution reduces to a noise-aware Leave no Trace, and we would say so.