The demonstration-guided reinforcement learning methods that work best on hard, sparse-reward manipulation tasks quietly assume a capability that real robots lack. Reinforcement learning (RL) means learning by trial and error from a reward signal. On these tasks, the strongest methods do not start every attempt from the task's normal starting configuration. Instead, they "teleport" the simulator directly into states taken from a human demonstration, often states close to the goal, and practice from there.
This proposal asks whether the teleport is doing essential work, or whether it can be replaced by simply driving the robot to those states with its own imitation policy. The core idea fits in two sentences. Run the current policy for the first $k$ steps of a demonstrated task, then switch to exploratory RL, and move $k$ earlier as the agent masters the later part of the task. The open question is how much of the benefit of teleporting survives when the robot must physically get to the practice state, and whether the answer is predicted by how far the roll-in drifts from the demonstration.
The answer matters for two reasons. First, it determines whether reset-based curricula can leave the simulator at all. Second, it connects a theoretical hierarchy of simulator access to a concrete engineering choice. At the top of that hierarchy is a full generative model that can be queried at any state. Below it are resets to previously visited states, then resets to demonstration states, and finally ordinary episodic resets only.
Core Claim
Reset access matters. Reverse Forward Curriculum Learning (RFCL) teleports the robot to points along each of a handful of demonstrations, starting near the goal and stepping backward as success rises. With 5 demonstrations it reaches roughly 39.4% success on ManiSkill2 PlugCharger under sparse reward, where RLPD and other baselines score 0%.
The theory says the same thing more sharply. The Power of Resets in Online Reinforcement Learning shows that resetting to previously visited states makes otherwise hard problems tractable. An Exponential Lower Bound for Linearly Realizable MDP with Constant Suboptimality Gap shows that losing reset access can cost exponentially many samples. Both results motivate this proposal, but neither covers demonstrations: RLLS assumes no demonstrator, and the lower bound assumes no expert.
Our claim is that a demonstration supplies most of what a reset supplies, as long as some policy can reliably reach demonstration states. Under ordinary episodic resets, an imitation policy rolled out for $k$ steps lands near the $k$-th demonstration state. That makes the landing state a genuinely visited state, so we are in the regime the reset theory analyzes, without a teleport.
The cost of this substitution has two parts. One is the environment steps spent on the roll-in. The other is the arrival error, meaning the gap between where the roll-in lands and where the demonstration was. Arrival error is a distribution-shift quantity. It grows quadratically in $k$ for behavior cloning in discrete settings and can grow exponentially in unstable continuous systems. Action chunking and noise-injected data provably control it under open-loop stability, as shown in Action Chunking and Data Augmentation Yield Exponential Improvements in Behavior Cloning for Continuous Spaces.
The prediction is therefore falsifiable. The performance gap between teleport curricula and roll-in curricula should track measured arrival error. Roll-in curricula should beat curriculum-free hybrid RL wherever arrival error stays small.
Research Program
We build a roll-in reverse curriculum on top of Imitation Bootstrapped Reinforcement Learning (IBRL). IBRL keeps a frozen imitation policy alongside an RL actor. At every step it executes whichever of the two proposed actions the critic scores higher, and it uses the same maximum when computing learning targets. We reuse that machinery unchanged so that any gain is attributable to the curriculum alone.
Each episode starts from the task's normal start distribution. For a chosen demonstration $i$ and switch time $k_i$, the agent runs the IBRL mixture actor greedily, with no exploration noise, for $k_i$ steps. This greedy phase is the roll-in. The agent then switches to exploratory RL until the episode ends.
All transitions from both phases go into the same replay buffer, so roll-in experience also trains the critic. Each demonstration keeps its own switch time, following RFCL's per-demonstration design so that conflicting demonstrations do not interfere. The curriculum starts with $k_i$ near the demonstration's end. Whenever success measured from the states actually reached passes a threshold, $k_i$ moves earlier. Once $k_i$ reaches zero for every demonstration, the method has become plain IBRL on the true task.
The roll-in policy is not fixed. As the critic improves, the mixture actor drives the roll-in, so roll-ins improve along with the policy. This is the mechanism that could let arrival error shrink over training, whereas a frozen behavior-cloning roll-in would not improve.
We log arrival error throughout training, measured as the distance in object and robot state between the landing state and demonstration state $k_i$. This logging is the analytical core of the project. It lets us place each task on a stability axis, using the contraction notion from the action-chunking analysis, and test whether that axis predicts when teleports are necessary.
Baselines
The naive reference point is behavior cloning on the same 5 demonstrations, plus RLPD-style RL with demonstrations mixed into the replay buffer. On PlugCharger, RFCL reports RLPD at 0% success with 5 demonstrations. Any method we propose that does not clear behavior cloning on the reset-free tasks counts as a failure.
The strongest reset-free competitor is IBRL itself, run without any curriculum. It is the most direct threat because it already keeps exploration near demonstrated behavior. It uses identical networks, demonstrations, environment-step budget (1M or 2M steps, matching RFCL's protocol), and hyperparameter search budget. We find no reported IBRL number for ManiSkill2, Meta-World, or robomimic in the file; the summaries give only learning curves. We will therefore run it ourselves on every task. Beating IBRL means higher final success at matched steps, or the same success in substantially fewer steps.
The strongest reset-dependent method is RFCL with true simulator teleports, reported at about 39.4% on PlugCharger. We treat RFCL as an upper reference, not a competitor to beat, because it uses access we deliberately deny ourselves. The headline metric is the fraction of RFCL's gain over IBRL that our method recovers. In our accounting, RFCL's teleports cost zero environment steps, while our roll-ins are charged in full.
Two ablations isolate the key ingredient. The first, "teleport-to-arrival," runs in the simulator. It teleports to states sampled from our roll-ins' empirical landing distribution but does not charge the roll-in steps. Comparing it with true RFCL isolates the cost of arrival error. Comparing it with our method isolates the cost of the roll-in steps themselves. The second ablation freezes the roll-in to the original behavior-cloning policy, which tests whether the improving roll-in matters.
Why not just…?
Why not just run IBRL, which already confines exploration to the demonstration manifold? This is the real threat. If IBRL alone solves PlugCharger and PegInsertionSide at matched steps, the curriculum adds nothing and we will report that. The reason to expect a gap is that IBRL still has to discover sparse reward from the start state. RFCL's backward curriculum exists precisely because that discovery fails on high-precision insertion, which is why RLPD sits at 0% there. The head-to-head comparison at matched steps settles the question.
Why not just replay the demonstration's actions open-loop to reach state $k$? Under the stability condition in the action-chunking paper, open-loop replay should land close to the demonstration state. On contractive tasks it may match learned roll-ins at no training cost. We include it as an additional roll-in variant. We expect it to break on contact-rich insertion, where that paper's smoothness assumption fails and small offsets in the start state compound. The arrival-error logs will show where it breaks.
Why not just train a separate reset agent, or do everything in a massively parallel simulator? Automating Reinforcement Learning With Example-Based Resets learns a reset controller from scratch, but only back toward a unimodal start distribution. It does not carry the robot forward to demonstration states. Emergent Dexterity Via Diverse Resets and Large-Scale Reinforcement Learning relies on programmatic resets across 65,000+ simulators, and its distilled real-robot policy still reaches only 25% on peg insertion. Our method needs neither a second agent nor a simulator, which is the point of the comparison.
Proposed Experiments
We will run the roll-in curriculum, IBRL, behavior cloning, RLPD, RFCL, and both ablations. The tasks are ManiSkill2 PlugCharger and PegInsertionSide, Adroit door and pen, and Meta-World Assembly and Box Close, all with 5 demonstrations, sparse reward, 5 seeds, and success rate against environment steps as the metric.
The decisive check is the regression of each task's performance gap to RFCL on measured arrival error. A strong relationship confirms the claim. A gap that stays large even when arrival error is near zero would mean the teleports' value lies elsewhere, most likely in the free roll-in steps. The teleport-to-arrival ablation identifies which explanation holds.
A secondary check injects controlled action noise into the roll-in to raise arrival error. If the claim is right, performance should degrade smoothly toward IBRL rather than collapsing.
If the simulator results hold, a final small trial will run the method on a physical arm with only episodic resets. The aim is to show it runs end to end, not to make a benchmark claim.
References
- Reverse Forward Curriculum Learning for Extreme Sample and Demo Efficiency
- The Power of Resets in Online Reinforcement Learning
- An Exponential Lower Bound for Linearly Realizable MDP with Constant Suboptimality Gap
- Action Chunking and Data Augmentation Yield Exponential Improvements in Behavior Cloning for Continuous Spaces
- Imitation Bootstrapped Reinforcement Learning
- Automating Reinforcement Learning with Example-based Resets
- Emergent Dexterity Via Diverse Resets and Large-Scale Reinforcement Learning