Robots trained in simulation are usually trained on many randomized versions of the world, with different friction, masses, and motor delays, so that the real world looks like just one more variation. This practice is called domain randomization, and it is the standard route to zero-shot sim-to-real transfer, meaning a policy that works on real hardware with no real-world fine-tuning. Two things break it on long manipulation tasks. The first is sparse rewards: the robot is told only "success" or "failure" at the end. The second is that the randomization range must include hard settings, such as slippery blocks or a heavy, sluggish arm. In those settings the robot almost never succeeds by chance, so it gets no learning signal at all. Meanwhile it keeps exploring wastefully in easy settings it has already mastered. Today every randomized world shares one exploration level, which is either a global entropy weight or a noise scale. That level is tuned to the average world and is wrong for most of them.
The proposal is to make exploration privileged. The simulator knows the exact physical parameters of every episode, even though the deployed policy never sees them. We use that knowledge to set exploration separately for each region of parameter space, driven by how often the robot currently succeeds there and how fast that rate is changing. Exploration is used only during training, so it can read simulator state freely without hurting transfer, much as asymmetric actor-critic methods let the training-only value estimator see privileged state. We study this in a data-scarce regime with a handful of demonstrations and binary terminal rewards, and test whether it widens the range of physics over which a single deployable policy succeeds.
Core Claim
Under sparse rewards and wide dynamics randomization, the exploration bottleneck is local in parameter space. A per-parameter exploration schedule that reads simulator parameters during training will reach higher success on the hardest parameter regimes than a single global schedule, at equal sample budget. This includes schedules that are already adaptive, such as SAC's automatic entropy tuning or the subtask-success-driven noise of A². The gain should come without lowering success on easy regimes, and it should survive transfer to real hardware. We also claim the effect does not reduce to curriculum over the randomization range. Curriculum changes which worlds the agent sees. Privileged exploration changes how the agent behaves inside each world.
Where This Sits
Four lines of work each hold one piece.
The first line motivates the gap. Tracking Drift (AES) adapts the entropy weight online and derives a rule in which exploration should scale with the square root of detected drift. It detects drift from spikes in temporal-difference error, which is the mismatch between predicted and observed value. Under binary rewards that signal is mostly zero, and it is global rather than per-world. Ready Policy One tunes its exploration weight from reductions in model error, but it needs dynamics ensembles that do not scale to image-based manipulation. A² is the closest method. It scales action noise by the robot's running success rate on each subtask and lifts SAC on PyBullet BlockStack from roughly 20% to roughly 90%. However, it runs only at fixed nominal physics. We reuse its idea that competence should drive noise, and move the index from "which subtask" to "which physics."
The second line supplies the environment side. Active Domain Randomization searches simulator parameters adversarially with a trajectory discriminator, and it transfers more reliably than uniform randomization on a real Poppy Ergo Jr. arm. Its policy-level exploration is static, though, and its discriminator signal degrades when every rollout earns zero reward. Our method is complementary: it acts on the policy's exploration, not on the sampling of worlds.
The third line, Informed Asymmetric Actor-Critic, establishes that training-only components may condition on privileged signals without biasing the policy gradient. It also shows that irrelevant privileged inputs act as "noisy TV" distractions. This constrains our design. We condition exploration only on the randomized physical parameters, never on raw simulator state, and we test whether filtering to the most return-relevant parameters matters.
The fourth line concerns dense progress critics. VLAC turns sparse rewards into dense learned progress signals and reports real-robot success rising from 30% to 90% in 200 episodes. The research file notes that such critics assume monotonic progress, and under randomized friction the robot must often retreat and regrasp. We therefore treat a progress critic as a competitor and a possible add-on, not as our exploration signal.
Research Program
The mechanism is deliberately small. During training we keep a running estimate of success rate $s(\xi)$ over the randomized parameter vector $\xi$, which includes friction, object mass, actuator gain, and latency. We store it as a kernel-smoothed table over normalized parameters, and we also keep a learning-progress estimate $\Delta s(\xi)$, the change in $s$ over a recent window.
For SAC, each episode drawn at parameters $\xi$ uses its own target entropy $\bar H(\xi)$, which is turned into a per-region temperature by the usual dual update:
$$\bar H(\xi) = H_{\min} + (H_{\max} - H_{\min})\,\sqrt{1 - s(\xi)}\,\big(1 + \beta\,|\Delta s(\xi)|\big),$$
Here $H_{\min}$ and $H_{\max}$ are the exploitation and exploration entropy targets. The factor $\beta \ge 0$ boosts exploration where competence is changing, which captures both rising competence and forgetting. The square-root form is borrowed from the AES drift rule, with failure rate standing in for drift. We treat that choice as a hypothesis and will compare it against linear and thresholded forms.
For PPO, or for pure action noise, the same quantity scales the noise standard deviation. The actor never receives $\xi$. At deployment it runs deterministically.
Data scarcity sets the regime. Each task gets 10 to 25 demonstrations collected at nominal physics. These seed the replay buffer and are injected A²-style, which also tells us whether demonstrations recorded in one world help exploration in distant worlds.
The analysis has two parts. The first is a map of success over $\xi$, showing where global schedules fail and whether privileged exploration fills those regions. The second is a stability check, because per-region temperatures make the off-policy buffer mix data from different exploration levels.
Baselines
The naive reference is SAC with its standard automatic global entropy tuning, trained under uniform domain randomization over the full target range, with the same demonstrations. Every competitor and our method gets the same environment steps, the same demonstrations, and the same randomization range. Each also gets the same hyperparameter search budget of 20 configurations per method, tuned on a held-out set of parameter settings. Results use 10 seeds.
The strongest prior competitor on this question is A² run under the same uniform randomization. It is the only method in the file that adapts exploration from competence under sparse multi-step rewards, and its reported number is about 90% BlockStack success versus about 20% for vanilla SAC at nominal physics (A²). We re-implement it with its 75% demonstration rate. Beating it means higher success averaged uniformly over the full parameter range, and specifically in the hardest quartile of parameters, with no loss in the easiest quartile.
AES is run with its default 0.9 quantile of TD-errors as the global adaptive-entropy competitor. The file reports SAC recovery time dropping from about 13.96% to 7.74% of training under abrupt drift (Tracking Drift), but no sparse-reward manipulation result. Active Domain Randomization is run as the environment-side competitor. For ErgoPusher the file states only that ADR had a tight success spread versus uniform randomization, so no reported numeric result was found on Lacuna for ADR on ErgoPusher.
Two ablations isolate the key ingredient. In the shuffled-index ablation, each episode receives a target entropy drawn from the same overall distribution our schedule produces, but assigned without regard to $\xi$. This tests whether conditioning matters, or whether a spread of exploration levels alone helps. In the global-competence ablation, the same formula is driven by overall success instead of $s(\xi)$. An oracle upper reference trains a separate policy for each parameter bin.
Why not just…?
Why not just rely on SAC's automatic entropy tuning? It is adaptive, but it enforces one entropy target averaged over all worlds. Easy worlds dominate successful data, so the average looks fine while hard worlds stay unexplored. The naive baseline tests this directly. If per-region success maps show no gap in hard regions for plain SAC, our premise is wrong.
Why not just use a curriculum that widens the randomization range as success grows? This is the real threat, because it attacks the same impasse by never presenting unreachable physics until the agent is ready. Our answer is that curriculum and privileged exploration act on different levers. Curriculum still runs one exploration level across all worlds currently in the range, and the final policy must still cover the full range. We run the curriculum alone, our method alone, and both together. If the curriculum alone matches our method on full-range success, we will report that privileged exploration adds nothing beyond curriculum and recast it as a complement only. The combination must beat both to support the core claim.
Why not just densify the reward with a learned progress critic such as VLAC? Dense progress makes exploration easier everywhere, but it does not decide where to explore. The file also documents that monotonic critics misread retreat-and-regrasp as regression, which randomized friction forces. We run a progress-shaped SAC baseline, and our method layered on top of it, on tasks that require recovery. If progress shaping alone closes the hard-regime gap, the exploration index is unnecessary in that setting.
Proposed Experiments
The main testbed is the PyBullet Multigoal suite from A² (ChestPush, ChestPick, BlockStack), with dynamics randomization added over block friction, block mass, gripper force, and actuator latency. We add ManiSkill pick-and-stack tasks for visual observations. For transfer we use the ErgoPusher setup from ADR and a real WidowX 250 S block-stacking task, the latter an arm used in DexScale. Metrics are success rate averaged uniformly over the full parameter range and over its hardest quartile, sample steps to 50% hard-quartile success, run-to-run variance, and real-robot success over at least 50 trials per condition, sweeping physical friction with taped and oiled surfaces.
Three checks are decisive. The method must beat A² under randomization in the hard quartile at equal steps. It must beat the shuffled-index ablation, which shows that conditioning on physics, not mere exploration diversity, drives the gain. The simulation advantage must carry over to real-robot success across friction settings. A further diagnostic tests a prediction: the gain should grow as the randomization range widens and shrink to zero at nominal physics, where our method should reduce to roughly A²'s performance. If the gain does not grow with range width, the per-region story is wrong even if average numbers improve.
References
- Abstract Demonstrations and Adaptive Exploration for Efficient and Stable Multi-step Sparse Reward Reinforcement Learning
- Tracking Drift: Variation-Aware Entropy Scheduling for Non-Stationary Reinforcement Learning
- Ready Policy One: World Building Through Active Learning
- Active Domain Randomization
- Informed Asymmetric Actor-Critic: Leveraging Privileged Signals Beyond Full-State Access
- A Generalist Pair-wise Progress Critic Model for Vision-Language-Action Robots
- DexScale: Automating Data Scaling for Sim2Real Generalizable Robot Control