When a robot learns a manipulation task by trial and error, it has to add random noise to its actions so that it tries new things, and that noise is normally turned down over training on a single global schedule. For tasks made of several stages, such as reaching an object, grasping it, carrying it, and inserting it into a slot, a single schedule is the wrong tool. Easy stages stay noisy long after the robot has mastered them, so it keeps dropping the object in transit and rarely reaches the hard stage. Meanwhile the hard stage loses its noise once overall success looks "good enough." We propose to give each stage its own noise level and to decay that level according to how reliably the robot completes that particular stage once it has entered it. The proposal is aimed at the data-scarce setting, where a few dozen to a few hundred human demonstrations both shape the noise and supply the stage boundaries.
The concrete starting point is Data-Guided Noise, from Reinforcement Learning via Implicit Imitation Guidance. That method replaces isotropic exploration noise with a state-dependent noise covariance learned from demonstrations. It anneals that covariance as the agent's aggregate task success rises, and it reports 2-3x gains over RLPD and IBRL on the hardest Adroit and Robomimic tasks. We keep its noise shape but change what drives the decay. In place of one scalar tied to whole-episode success, the scale is tied to stage-conditional success, and it can rise again if a stage starts to fail. The claim is narrow and testable. On multi-stage tasks whose stages differ sharply in difficulty, stage-local decay should reach the bottleneck stage more often and solve it with fewer environment steps. On single-stage tasks it should make no difference.
Core Claim
The global signal is a poor proxy for multi-stage tasks. Whole-episode success is roughly the product of per-stage completion rates, $p = \prod_k p_k$, so it stays near zero until every stage works and then rises quickly. A schedule driven by $p$ fails in two ways. It keeps noise high on stages the agent has already mastered, which is exactly the regime where Action Noise in Off-Policy Deep Reinforcement Learning: Impact on Exploration and Performance found that more noise truncates episodes and lowers return. It then shrinks noise on the bottleneck stage as soon as the product climbs. We instead execute
$$a_t = \pi_\theta(s_t) + \epsilon_t,\qquad \epsilon_t \sim \mathcal{N}\big(0,\ \beta_{k(s_t)}\,\Sigma_\phi(s_t)\big),\qquad \beta_k = \max\big(\beta_{\min},\ \beta_0 (1-\hat p_k)^{\alpha}\big).$$
The symbols are as follows. $\pi_\theta$ is the reinforcement-learning actor. $\Sigma_\phi(s)$ is the demonstration-derived covariance from Data-Guided Noise. $k(s_t)$ is the stage the robot is currently in. $\hat p_k$ is the fraction of recent episodes that entered stage $k$ and completed it, estimated over a sliding window so that it can fall as well as rise. $\beta_0$, $\beta_{\min}$ and $\alpha$ are scalar schedule constants. The covariance shape stays exactly as Data-Guided Noise learns it; only its magnitude is reallocated across stages.
Research Program
Stage labels come from structure that data-generation pipelines already produce. MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations splits each human demonstration into object-relative subtask segments in order to transform them across scenes. We reuse those segment boundaries and the simulator's subtask-completion predicates to label stages, both in the demonstrations and during online rollouts. MimicGen is a source of method here, not a competitor: we borrow its segmentation and discard its open-loop replay.
Because privileged predicates are unavailable on real hardware, a second variant trains a small stage classifier on the segmented demonstrations, using proprioception and, in the pixel setting, image features. That classifier is then applied to online states. Comparing the predicate-labelled and classifier-labelled versions tells us how much of any gain survives imperfect stage detection.
The actor-critic backbone follows RLPD from Efficient Online Reinforcement Learning with Offline Data. RLPD seeds the replay buffer with demonstrations, samples half of every batch from them, and adds LayerNorm and a critic ensemble so the critic does not overestimate unfamiliar actions. Using it means every method compared shares the same critic and replay machinery and differs only in exploration.
Two prior works also modulate exploration by demonstration structure. Constrained movement primitives, in Guiding reinforcement learning using constrained uncertainty-aware movement primitives, widen noise where demonstrations vary and narrow it where they agree. Demonstration-guided Soft Actor-Critic, in Exploration-efficient Deep Reinforcement Learning with Demonstration Guidance for Robot Control, injects noisy expert actions when the agent is near a demonstration in Euclidean state space. Neither one decays guidance per stage. The first relies on hand-specified constraints and analytical primitives. The second's distance check breaks down in high dimensions, and its noise is isotropic.
We deliberately do not extend this to diffusion-policy sampling temperature, the second gap the literature leaves open around Diffusion Policy Policy Optimization. Keeping the policy class fixed isolates the scheduling question.
The analysis tracks three quantities per stage throughout training: the rate at which each stage is entered, the stage-conditional completion rate, and the fraction of episodes that end before the bottleneck stage is reached. The mechanism predicts that stage-local decay sharply reduces those early truncations. That mediating prediction can be checked directly, separately from final success.
Baselines
The naive reference point is RLPD with its standard entropy-driven exploration and no demonstration-shaped noise, run from Efficient Online Reinforcement Learning with Offline Data. It shows what critic stabilization alone achieves. The file quotes only a relative figure for RLPD, a "2.5x improvement over existing baselines" on Adroit Door, and no absolute success rate on our tasks, so we rerun it.
The strongest existing method is Data-Guided Noise with its original success-rate annealing, from Reinforcement Learning via Implicit Imitation Guidance. It is the closest competitor because our method differs from it only in the decay signal. Its reported result is relative: "2-3x improvement in sample efficiency and final success rates on the most difficult tasks compared to … RLPD and IBRL." No absolute Data-Guided Noise success number on Robomimic or RoboCasa was found on Lacuna, so we reimplement it inside the same backbone.
We also run IBRL from Imitation Bootstrapped Reinforcement Learning. At each step it executes whichever of a frozen imitation policy's action or the RL actor's action the critic scores higher. It is the strongest representative of handling demonstrations through the action mean rather than the noise.
As an imitation-only ceiling, we include BC-RNN, a behaviour-cloning policy with a recurrent memory, from What Matters in Learning from Offline Human Demonstrations for Robot Manipulation. That paper reports 84% success on Square and 71.3% on Transport with proficient-human data. For calibration on Transport we cite DPPO's reported ">90% success rate from state observations," while noting that it uses a different policy class and a much larger on-policy interaction budget.
Two ablations isolate the key ingredient. The first applies the same demonstration-shaped covariance with a global linear time decay; it asks whether any decay schedule suffices. The second is a stage-free per-state scale set by disagreement among the critic ensemble that RLPD already trains; it asks whether stages matter at all, or only locality does.
All methods share the same demonstrations, the same environment-step budget, the same critic architecture and update-to-data ratio, and the same hyperparameter sweep of equal size. Each runs with five or more seeds. Beating Data-Guided Noise means a statistically separated improvement, either in steps to reach its final success or in final success at a fixed budget, on at least two multi-stage tasks. It also means no loss on the single-stage controls.
Why not just…?
Why not just keep Data-Guided Noise's global annealing and tune it better? Tuning a single scalar cannot fix the product structure of whole-episode success. Any global level is simultaneously too high for mastered stages and too low for the bottleneck. The experiment settles this directly: the global variant receives the same tuning budget, and we measure per-stage truncation. If the tuned global schedule matches stage-local decay on the most imbalanced tasks, the claim is false.
Why not just use a linear time decay, as DrQ-v2 and TD3 do? Action Noise in Off-Policy Deep Reinforcement Learning: Impact on Exploration and Performance shows linear and logistic decay beat constant noise. Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning uses linear decay successfully from pixels. A time schedule, however, cannot react when a late stage remains unsolved after noise has run out. The linear-decay ablation, which keeps the demonstration-shaped covariance, quantifies how much the reactive, re-heatable signal adds.
Why not just scale noise by critic-ensemble disagreement, with no stages at all? This is the real threat. Ensemble disagreement is free in RLPD and is also local in state, so it might recover most of the benefit without stage labels. We run it as a full baseline. If it matches stage-local decay, the honest conclusion is that locality, not stage structure, is what matters, and we would report that. We expect disagreement to fall wherever the critic has seen data, regardless of whether the stage is actually solved. The per-stage completion curves will show whether that happens.
Why not just generate more demonstrations with MimicGen and skip reinforcement learning? RoboCasa: Large-Scale Simulation of Household Tasks for Generalist Robots shows synthetic augmentation lifts atomic-task success from 28.8% to 47.6%, yet composite long-horizon tasks stay "often below 12%." Open-loop replay cannot correct contact errors. We include a behaviour-cloning-on-MimicGen-data baseline on the RoboCasa composite tasks to test whether closed-loop stage-local exploration exceeds it.
Proposed Experiments
The main suite uses Robomimic Square, Transport and Tool Hang with the proficient-human data. We run each at 200 demonstrations and at a subsampled 50, to stress scarcity. We add a handful of RoboCasa composite tasks, chosen because they have the most stages and the lowest reported success. Adroit Door and Robomimic Lift serve as single-stage controls, where we predict no difference. The metrics are task success versus environment steps, per-stage completion and entry rates, and the fraction of episodes that end before the bottleneck stage.
The decisive check is a difficulty dial. In Square we progressively tighten the insertion tolerance while leaving the reach, grasp and transport stages unchanged. If stage-local decay is right, its advantage over global annealing should grow as the stage imbalance grows, and that growth should coincide with fewer pre-bottleneck truncations. If the advantage is flat, or appears equally in the ensemble-disagreement baseline, the stage-structure hypothesis is rejected.
A final pixel-input run uses the learned stage classifier on Square. It tests whether the gain survives imperfect stage labels, which is the step a real-robot deployment would require.
References
- Reinforcement Learning via Implicit Imitation Guidance
- Action Noise in Off-Policy Deep Reinforcement Learning: Impact on Exploration and Performance
- MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations
- Efficient Online Reinforcement Learning with Offline Data
- Guiding reinforcement learning using constrained uncertainty-aware movement primitives
- Exploration-efficient Deep Reinforcement Learning with Demonstration Guidance for Robot Control
- Diffusion Policy Policy Optimization
- Imitation Bootstrapped Reinforcement Learning
- What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
- Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
- RoboCasa: Large-Scale Simulation of Household Tasks for Generalist Robots