Probing the Edge of Recoverability Without Resets

A robot that learns on real hardware cannot be teleported back to a clean start after every attempt. Some method has to decide, moment to moment, whether the next action is one the robot can later undo. The standard answer, from Leave no Trace, is to train a second "reset" policy alongside the task policy and to abort any forward action after which the reset policy doubts it could get home. That abort rule is only as good as the model that predicts recoverability, and that model is trained on the robot's own experience. Our central observation is that this setup inherits a failure the safe-RL community found in the episodic setting and called the safety paradox (Breaking Safety Paradox with Feasible Dual Policy Iteration). The better the agent gets at avoiding irreversible states, the fewer near-irreversible transitions it sees, and the less its recoverability model knows about exactly the region it most needs to judge. A second problem compounds the first. The actions a task actually hinges on, such as inserting a peg or pushing an object close to a table edge, are rare in a reset-free data stream. The recoverability model is therefore uncertain about them because it has little data, not because they are dangerous, and a pessimistic abort rule vetoes them anyway.

We propose recoverable boundary probing. A third, short-lived probing policy deliberately steers the robot into the band of states the abort rule currently forbids, but only where a stricter hard floor on predicted recoverability still holds. Its goal is to collect graded "near-miss" data: transitions where the reset policy's chance of success drops sharply but does not reach zero. The existing way to fix the safety paradox is to crash on purpose and rely on an episodic reset. Here the reset policy's own success probability serves as a continuous safety signal, so boundary data can be gathered without ever needing a human to reset the scene. Because the band sits where the recoverability ensemble disagrees, and that disagreement concentrates on rare actions, the same probing both repairs the model's calibration near hazards and stops it from vetoing rare but necessary task actions.

Core Claim

In reset-free learning with irreversible states, a recoverability model trained only on the forward and reset policies' own data becomes miscalibrated in two linked ways. It is optimistic near true sink states, because failure data gets scarce as the agent improves. It is pessimistic on rare task actions, because it has little data about them. Both errors can be reduced at once, with fewer manual resets than either conservative aborting or deliberate crashing, by actively sampling the band between an abort threshold and a hard recoverability floor.

Positioning Against Prior Work

The reset-free line supplies our base machinery. Leave no Trace equates safety with reversibility and aborts when any member of an ensemble of reset-policy value functions predicts failure. It reported a 2.5x reduction in manual resets on Pusher and near-zero resets on Cliff Cheetah. It treats every reset as the same binary cost and never asks whether its pessimism blocks useful rare actions. Intelligent Switching for Reset-Free RL (RISC) contributes a learned "success critic," a model of the probability of reaching a target, which we reuse as our graded recoverability signal. RISC also contributes bootstrapping that does not zero out values when control passes between policies. RISC itself has no cost signal and assumes every state can be reached again, which is the assumption we drop. Reset-Free Lifelong Learning with Skill-Space Planning avoids sink states by planning only over learned skills, which keeps safety implicit. It also shows that plain Soft Actor-Critic collapses once resets are removed, which rules out naive continuing RL as a baseline worth tuning. The EARL benchmark paper (Autonomous Reinforcement Learning: Formalism and Benchmarking) motivates the whole problem. It finds that on Sawyer-Peg and Franka-Kitchen "all autonomous algorithms failed to make significant progress," and it identifies non-ergodic traps as the open problem.

The safe-exploration line supplies both the diagnosis and the constraint on our design. FDPI (Breaking Safety Paradox with Feasible Dual Policy Iteration) proves that violation data starves as policies become safe. Its fix pairs the safe policy with a reckless dual policy, kept close to it by a KL penalty, that maximizes violations. With importance-weighted training of the feasibility model, this keeps roughly 10x more violation samples in late training. FDPI motivates our method: it shows active sampling beats passive reweighting. It also constrains the method, because every violation it harvests ends an episode, which on hardware means a human reset or a broken part. We keep FDPI's proximity constraint and importance correction. We replace "maximize violations" with "maximize the drop in recoverability, subject to a floor." Recovery RL hands control to a recovery policy when a risk critic fires and reports 2 to 20 times better success-versus-violation trade-offs. However, it pretrains that critic on an offline failure dataset and relies on episodic resets. Our method is meant to build the equivalent of that dataset online and without resets. Constrained Markov Decision Processes via Backward Value Functions gives feasibility during training through conservative state-wise bounds. It admits a suboptimality gap, and we expect that gap to be worst on rare actions. It serves as a conservative comparison point rather than a component.

The long-tail line serves as our lens and evaluation standard rather than our mechanism. Zipfian Environments for Reinforcement Learning shows the "illusion of mastery": near-perfect average scores alongside failure on more than 40% of rare items. It also shows that prioritized replay recovers only 17% for R2D2. Playing Minecraft with Behavioural Cloning shows that a single missed rare bottleneck action causes compounding failure, and that hand-coding that action lifted reward by 30% to 92%. We borrow their practice of reporting performance on the rare slice separately. We also borrow the hypothesis that rare actions, not average competence, set the ceiling.

Research Program

The system has three policies sharing one continuous lifetime. The forward policy $\pi_f$ pursues the task. The reset policy $\pi_b$ returns to a set of start states. An ensemble of $K$ success critics $F^k(s)$ estimates the probability that $\pi_b$ reaches the start set from state $s$. We write $\bar F(s)$ for the ensemble mean and $L(s)$ for a lower confidence bound (the mean minus a multiple of the ensemble standard deviation). As in Leave no Trace, the forward policy is aborted whenever $L(s') < \tau_{\text{abort}}$ for the predicted next state $s'$.

The addition is the probing policy $\pi_p$. It is invoked occasionally, from states the forward policy has reached, and runs for a few steps before handing control to $\pi_b$. It solves

$$\max_{\pi_p}\; \mathbb{E}\big[\,\bar F(s_t) - \bar F(s_{t+1})\,\big] \quad \text{s.t.}\quad L(s_{t+1}) \ge \tau_{\text{hard}},\;\; \mathrm{KL}\big(\pi_p(\cdot\mid s_t)\,\|\,\pi_f(\cdot\mid s_t)\big) \le \delta,$$

where $\tau_{\text{hard}} < \tau_{\text{abort}}$ is a hard floor on recoverability and $\delta$ bounds how far probes stray from task behavior. In words, the probe tries to lower the reset policy's chance of success as much as possible without letting the pessimistic estimate fall below the floor, and it stays near actions the task policy would plausibly take. It therefore spends its time in the band $\tau_{\text{hard}} \le L < \tau_{\text{abort}}$, which the forward policy is never allowed to enter. That band has high ensemble disagreement by construction. We predict it is disproportionately populated by rare task actions.

Every probe ends with $\pi_b$ attempting recovery. The outcome, success or the number of steps taken, is a graded label for training $F^k$. Training uses FDPI-style importance weights so the forward policy's value estimates are not biased by probe data. We also record probe-derived labels separately, so we can measure how much of the calibration gain comes from them.

Two further analyses make the claim testable. First, in simulation we can save a state and run many reset-policy rollouts from it to compute ground-truth recoverability. That lets us measure calibration error of $\bar F$ as a function of true recoverability and of how rare the preceding action was. Second, we sweep $\tau_{\text{hard}}$ to trace the frontier of manual resets against critic calibration. This includes the point $\tau_{\text{hard}} = \tau_{\text{abort}}$, where probing is disabled, and a limit where the floor is removed, which recovers FDPI-in-reset-free.

Baselines

The naive reference point is Soft Actor-Critic run continuously with no reset policy, since LiSP reports that it "crashed to zero reward almost immediately after resets were disabled". It anchors the manual-reset count from above. The primary incumbent is Leave no Trace with its pessimistic ensemble abort. It is the most direct competitor because it already targets both resets and irreversibility, and its reported result is a 2.5x reduction in manual resets on Pusher and near-zero resets on Cliff Cheetah. Beating it means fewer lifetime manual resets at equal or better deployment success, together with higher success on the rare-action slice.

We also run a strengthened Leave no Trace that uses RISC's success critic and non-terminal switching. That isolates probing from simply having a better critic. For RISC, the file reports only that it "matched or exceeded the best-known results on 3 out of 4 EARL tasks", with no per-task numbers found on Lacuna.

FDPI transplanted into the reset-free setting is the strongest active-sampling competitor: its dual policy crashes, and each crash that reaches a sink state is charged as a manual reset. FDPI reports near-zero violations across 14 Safety-Gymnasium environments and a roughly 10x higher violation-sample ratio than SAC-FPI in late training, both under episodic resets. Beating it means matching its critic calibration with markedly fewer manual resets.

Recovery RL is run in two forms. One is given the offline failure data it expects. The other is given none, which shows the cost of lacking that data. SAC-Lagrangian gives a standard constrained-optimization point.

The key ablation replaces the probe objective with random perturbations inside the same band. This tests whether targeted descent on recoverability matters or whether simply visiting the band is enough. All methods share network sizes, the same number of environment steps, and an equal hyperparameter-search budget per method. Abort thresholds are tuned for Leave no Trace with the same budget as $\tau_{\text{hard}}$ is tuned for ours.

Why not just…?

Why not just lower Leave no Trace's abort threshold or shrink its ensemble pessimism? This is the most serious alternative. Loosening the abort rule also lets the agent into the band and would let rare actions through. The difference is that it lets the task policy in everywhere, with no floor, and it does not aim data at the states where the critic disagrees. Our threshold sweep settles the question directly. If a tuned threshold matches our reset-versus-rare-success frontier, the probing policy is unnecessary and we will report that.

Why not just oversample rare and near-failure transitions with prioritized replay? Reweighting cannot create data that was never collected. FDPI shows violation samples fall below 1% of the buffer as policies become safe, and in Zipfian environments prioritized replay recovered only 17% for R2D2 while more than 40% of rare items still failed. We nonetheless include prioritized replay on the strengthened Leave no Trace as a baseline, because a reset-free buffer may retain more near-misses than an episodic one.

Why not just run FDPI and pay for the resets? If the reckless probes cost only a handful of human resets early in training, this could be the cheaper engineering answer. The reset-free FDPI baseline measures that cost directly, in manual resets per unit of calibration gain. Our claim fails if FDPI's reset count is comparable to ours.

Why not just use a constrained-optimization method such as Lagrangian SAC? Backward value functions and Recovery RL both report that Lagrangian methods satisfy constraints only at convergence. Safety-Gymnasium reports oscillation between recklessness and stalling. In a lifetime where one violation can end learning, those transient violations become permanent sink states. The Lagrangian baseline quantifies this.

Proposed Experiments

No existing suite combines reset-free execution, explicit hazards, and a skewed action structure, so we build one from existing parts. From the non-episodic MuJoCo tasks of Leave no Trace we take Pusher, Cliff Cheetah and Peg Insertion. From EARL we take Tabletop-Organization and Sawyer-Peg. To each we add irreversible zones: an edge region where objects become unrecoverable, and joint-limit regions whose violation freezes the scene until a counted manual reset. We impose a long tail by making task success depend on a rare action class, placing or inserting near a hazard, whose frequency in successful trajectories we control on a Zipfian schedule. We also convert two Safety-Gymnasium navigation tasks (Goal and Push) to reset-free form, where a violation creates a sink state rather than ending the episode.

Metrics are lifetime manual resets, lifetime cost violations, and deployment success from canonical starts, all following EARL's evaluation protocol. We add success on the rarest 20% of action or object configurations, following the Zipfian protocol, and calibration error of the success critic against simulator ground truth, reported separately for common and rare preceding actions.

Three checks are decisive. First, without probing, the recoverability critic's calibration near true sink states must degrade late in training, which would confirm the safety paradox carries over to reset-free learning. If it does not degrade, the premise fails. Second, probing must reduce calibration error on rare-action states more than on common ones, confirming that the band concentrates rare actions. Third, the full method must dominate both tuned Leave no Trace and reset-free FDPI on the joint axes of manual resets and rare-slice success, and the random-perturbation ablation must fall between them. Sawyer-Peg is the hardest test: EARL reports that autonomous methods there make no significant progress, so even partial rare-slice success would be informative, and failure there would bound the claim to simpler contact tasks.