Learning Which Mistakes Need a Human: Budgeting Manual Resets in Reset-Free Robot Learning

A robot that learns by trial and error on real hardware needs a person to fix the scene whenever it makes a mess it cannot undo. The robot might drop a peg off the table, jam a drawer, or knock a part out of reach. "Reset-free" methods train a second controller to undo the first one's effects, but they assume every mess can be undone. When a mess cannot be undone, someone still has to walk over, usually without warning. We propose to treat those unplanned human rescues as the scarce resource. The robot learns, during training itself, a small set of human-readable rules, called predicates (true/false tests on the scene such as "peg center is past the table edge"). These rules pick out the states its own reset controller cannot recover from. The robot then limits how often it risks entering such states. When a risky experiment is worth running, it asks a person in advance to stand by, rather than failing and waiting to be found.

The claim is that learning irreversibility as symbolic rules, rather than as a black-box neural score, lets the robot learn its own points of no return from a handful of costly failures. The same rules should carry over to scenes with more objects. Scheduling risky exploration into short, announced supervision windows should then cut total human effort compared with three alternatives: fixing things only after they break, forbidding risk outright, or learning irreversibility with a neural classifier.

What the four directions contribute, and where the gap is

Reset-free reinforcement learning (RL) already has good machinery for reversible mistakes. A State-Distribution Matching Approach to Non-Episodic Reinforcement Learning (MEDAL) trains a backward controller to return the scene toward states seen in 10–30 demonstrations. Intelligent Switching for Reset-Free RL (RISC) decides when to switch between the forward and backward controllers using a learned estimate of how likely the robot is to succeed from where it is. Both papers motivate this proposal through what they report on irreversible states. Earlier reset-free baselines scored 0% on Sawyer Peg Insertion because pegs fell off the surface, and MEDAL reached only 40%. The most-cited open problem in this line is the "sink state" from which recovery is impossible. Current methods either never see such states or silently stall in them.

Safe exploration supplies the constraint machinery but assumes episodes, which makes it a constraint on our design rather than a ready-made tool. Constrained Markov Decision Processes via Backward Value Functions turns a trajectory-level cost budget into a per-state test by tracking cost spent so far. It relies on the budget clearing at each episode reset. Safe Exploration via Policy Priors (SOOPER) falls back to a safe prior controller when the budget is at risk, and Neurosymbolic Reinforcement Learning with Formally Verified Exploration needs a known worst-case dynamics model; both also assume episodes. Most relevant is Adaptive Probabilistic Shielding by Learning MDPs for Safe Reinforcement Learning. It bounds unknown transition probabilities with confidence intervals and shows that some unshielded exploration is necessary: without it, the agent never gathers the data to prove an action safe. That result creates a tension we must resolve. Our resolution is to confine unshielded exploration to moments when a human has been warned and is watching.

Domain modeling supplies the method. Predicate Invention for Bilevel Planning searches a grammar of feature inequalities, negations and quantifiers for predicates that make planning efficient. Its learned predicates carry over to scenes with 2–3x more objects. Continual learning and refinement of causal models through dynamic predicate invention repairs a symbolic transition model online by generalizing or specializing rules after each prediction error. It masters MiniHack Lava Crossing after a single failure, where PPO needed about 128 episodes for first success. Both assume deterministic, fully observed dynamics, and neither aims its predicates at recoverability. We reuse their grammar and their predict-verify-refine loop, but point them at a new target. The target is whether the robot's own backward controller can undo a state, and we attach probabilistic counts to handle noise.

Proactive human-robot interaction supplies the interaction regime, with less weight than the other three. That literature usually means anticipating a human's goal (Fast Motion Prediction for Collaborative Robotics; ProAct). We borrow its central observation: anticipating an event beats reacting to it after the fact, because reacting causes stop-and-go delays and extra human effort. We turn that observation toward the robot's own future need for help. Proactive Human-Robot Co-Assembly provides a hand-specified geometric safety layer (the Safe Set Algorithm over human skeletal keypoints). We use it as-is whenever a person is nearby. It is infrastructure, not a contribution.

The closest existing systems are Tether (Autonomous Functional Play with Correspondence-Driven Trajectory Warping) and the reset-free RL methods above. Tether reached 5 interventions in 26 hours by chaining tasks whose end states set up the next task and replaying warped demonstrations. It does not explore beyond demonstration-like motions, does not learn what is irreversible, and has no way to anticipate needing a person. Our setting is RL that genuinely explores. There, irreversible events are the main source of human burden.

Core Claim

Define the robot-relative irreversibility cost $c_t = 1$ if the state reached at time $t$ cannot be returned to the demonstration state distribution by the current backward controller within $H$ steps. Otherwise $c_t = 0$. The robot must learn from a single continuing stream, so we constrain the long-run rate of unannounced human rescues rather than a per-episode budget:

$$\limsup_{T\to\infty} \frac{1}{T}\,\mathbb{E}\Big[\sum_{t=1}^{T} c_t\,(1-w_t)\Big] \le \epsilon, \qquad \text{minimize}\;\; \frac{1}{T}\sum_{t=1}^{T} \kappa\, w_t .$$

Here $w_t \in \{0,1\}$ indicates that a supervision window the robot requested in advance is active, and $\kappa$ is the per-step cost of human attention during a window. The task-reward objective of the reset-free learner is unchanged. The claim has three parts. First, a symbolic irreversibility model learns this cost from far fewer irreversible events than a neural classifier and transfers to more objects. Second, a shield that blocks actions whose upper-bound irreversibility risk would break the rate constraint keeps unannounced rescues near $\epsilon$. Third, letting the robot buy windows proactively for high-value, high-risk experiments yields better task success per unit of total human effort than any of the alternatives.

Research Program

The system has three components on top of a RISC learner. We choose RISC because its success estimate already provides a frontier signal.

The first component is an irreversibility predicate learner. Each transition is labeled after the fact. If the backward controller later fails to recover within $H$ steps, or a human must step in, the preceding transition is marked irreversible. Near-misses, where recovery took unusually long, receive soft labels. Over object-centric features (poses, contacts, positions relative to surface edges and containers), we search the grammar of Predicate Invention for Bilevel Planning. The scoring objective is the likelihood of the irreversibility labels plus a complexity penalty, replacing that paper's planning-time objective. When a new irreversible event is unexplained, or a predicted one does not occur, the rules are generalized or specialized as in Continual learning and refinement of causal models through dynamic predicate invention. To handle stochastic outcomes, each abstract state defined by the predicates keeps Beta counts of irreversible transitions per action class. Following Adaptive Probabilistic Shielding, we use a confidence upper bound, so rules are never treated as certain.

The second component is a rate shield. A running estimate of the unannounced-rescue rate plays the role of cost-so-far, in the spirit of Backward Value Functions but over a sliding time window rather than an episode. When a proposed action's upper-bound risk would push the windowed rate above $\epsilon$, the action is replaced by the backward controller or the best action below threshold. This is SOOPER-style fallback without episodes.

The third component is a proactive supervision scheduler. Blocked actions are queued together with their estimated information value: the predicted change in the frontier success estimate, plus the narrowing of the risk interval. When the queue's summed value exceeds $\kappa$ times the expected window length, the robot requests a window with a stated duration. It then drives the scene to the abstract state where the queued experiments start and runs them without the shield while the person watches. The rescue that follows costs only scheduled attention. Because the predicates are readable, the request can say what the robot is about to try ("testing pushes near the table edge"). This is the proactive-interaction element: the robot anticipates its own need for help instead of being discovered stuck.

Baselines

The naive reference point is forward-backward RL on fixed switching schedules with no shield. Every irreversible event triggers an unannounced rescue. It anchors the rescue count and matches the setups that MEDAL reports at 0% on Sawyer Peg Insertion.

The strongest existing reset-free methods are MEDAL and RISC run as published. For MEDAL the reported figure is 40% autonomous success on Sawyer Peg Insertion. For RISC, the file reports matching or beating the best known results on 3 of 4 EARL tasks, but no reported per-task success number was found on Lacuna. We will rerun both from their released settings, with the same demonstrations (10–30), environment steps and tuning budget as our method. Beating them means equal or higher final success with at least a 2x reduction in unannounced rescues per 100k steps.

The strongest competitor on the actual question is RISC plus a neural irreversibility shield. The same labels train an ensemble classifier on the same features and images, which feeds the identical rate shield and scheduler. This ablation isolates the key ingredient, the symbolic predicate model. It gets the same label stream, compute and hyperparameter search budget as our method. Beating it means fewer irreversible events before the rescue rate drops to $\epsilon$, and a lower rescue rate when test scenes contain 2–3x more objects.

Two further ablations isolate the human-facing ingredient. Pure shield, with no windows, tests the conservative trap described by Adaptive Probabilistic Shielding. Reactive supervision lets the robot run unshielded whenever its risk is high and call a human only after getting stuck. This tests whether calling in advance matters. Finally, an oracle predicate variant uses hand-written irreversibility rules (such as "object off surface") to bound how much the predicate learner loses relative to manual domain modeling.

Why not just…?

Why not just add a control barrier function or geometric safety filter to RISC or MEDAL? Such filters, like the Safe Set Algorithm in Proactive Human-Robot Co-Assembly, enforce distance constraints that are specified by hand. The failures that stall reset-free learning are mostly not collisions. A peg rolling off an edge is geometrically harmless and still needs a human. We keep a geometric filter for human proximity, but it cannot supply the irreversibility cost. The oracle-predicate ablation shows how much of the gain comes from knowing which states are irreversible.

Why not just learn a neural reversibility classifier, or reuse RISC's success estimate? This is the real threat. Our bet is that irreversible events are rare and each costs a human rescue. A compact rule over edge and contact features should become accurate after a few events and carry over to more objects, as symbolic learners did in Continual learning and refinement of causal models through dynamic predicate invention and Predicate Invention for Bilevel Planning. A neural classifier may need many more events. The matched neural-shield baseline settles this directly. If it reaches the same rescue rate within 1.5x as many irreversible events and transfers equally well, we drop the predicate claim, and the contribution reduces to the rate shield and the scheduler.

Why not just penalize irreversible states in the reward? Penalty tuning has two failure modes, documented in Safety Gymnasium and Backward Value Functions. A penalty that is too small accepts failures, and one that is too large freezes the robot. A penalty also cannot tell a supervised experiment from an unsupervised one. We include a penalty-swept RISC variant with three penalty scales to confirm this on our tasks.

Why not just let a person watch and fix things as needed, as Tether did? Tether achieved 0.26% interventions by staying close to demonstrated motions, not by exploring. For exploring RL, the reactive-supervision ablation is exactly this option. If it matches our method on total human cost at equal success, then calling in advance adds nothing and we will say so.

Proposed Experiments

Development and sanity checks use MiniHack Lava Crossing and LiSP's Volcano domain (Reset-Free Lifelong Learning with Skill-Space Planning), both of which have explicit permanent sinks. There, the predicate learner should recover the lava or sinkhole rule after one to a few events. The main evaluation uses the four EARL tasks (Tabletop, Sawyer Door, Sawyer Peg Insertion, Minitaur). We add variants of Peg Insertion and Tabletop with more objects and an elevated shelf, to create more irreversible states and to test transfer across object counts.

In simulation, a "human" is modeled as a teleport-back rescue with two costs. An unannounced rescue incurs a fixed disruption cost plus the time until it is noticed, drawn from a delay distribution. A scheduled window costs $\kappa$ per step with no delay. We report three quantities: task success over training; unannounced rescues per 100k steps; and total human cost under three values of $\kappa$, so that no single cost ratio decides the conclusion. We run 10 seeds and give confidence intervals.

A physical follow-up on a tabletop arm, modeled on the Tether task set and gated on the simulation results, measures real human minutes and intervention counts over a 10-hour run. Tether's 5 interventions in 26 hours serves as the reference point, not as a like-for-like comparison.

The decisive checks are three. The first compares predicate and neural shields: plot irreversible events experienced against the rescue rate achieved, including on scenes with more objects. The second compares proactive windows, pure shield and reactive supervision on the success-versus-total-human-cost frontier. If the pure shield's success plateaus below the proactive variant's, that confirms the conservative trap; if reactive supervision matches ours, the proactive claim fails. The third measures how often the learned predicates agree with the oracle rules, and whether the rules hold up when transitions are made stochastic by adding friction noise. This directly tests the determinism limitation shared by both predicate-invention papers.