A robot trained in simulation under domain randomization learns to act across many randomly drawn versions of the physics, with friction, mass and damping all varied. It then faces one particular physical world whose parameters it does not know. When the task has a sharp success criterion, such as sliding a puck into a narrow zone, striking an object to a target pose, or seating a peg, the robot has no way to learn which world it is in. It therefore hedges, and the hedge is often a policy that succeeds nowhere. The sparse success signal hides something useful: every failed attempt ends in an outcome, like where the puck stopped or how far the object slid, and that outcome is a measurement of the physics. We propose training, entirely in simulation, a policy whose input is the record of its own previous attempts on the same physical instance and their outcomes. It learns to turn early misses into corrections, and it adapts on the real robot without any weight updates.
The training problem is hard in exactly the way the sparse-reward literature describes. A policy that must learn both to act and to read its own failures gets a binary reward only at the end of a multi-attempt episode. We make this tractable by combining two tools. The first is hindsight relabeling, which pretends that wherever an attempt ended was the intended goal. The second is a training-only critic (the value estimator used during learning) that sees the true simulator parameters. The claim is specific: for short, contact-driven tasks where the outcome of one attempt reveals the physics, this in-context adaptation over attempts closes most of the gap between randomized training and a policy that knows the true parameters. It does so without an explicit identification step, without retraining in a calibrated simulator, and without the continuous proprioceptive stream that online-adaptation methods need.
Where the gap is
The failure this work targets is well documented. Several papers show that uniform randomization produces a "lowest common denominator" policy that is safe but imprecise. MANGA describes a conservative, average policy. Domain Contraction reports a final-state error of 0.653 for reinforcement learning with randomization against 0.015 for its contracted policy on an open-loop Hit primitive. ASID reports that plain randomization reaches 0/9 on rod balancing and 30% on shuffleboard on a Franka arm. These results motivate the proposal.
The existing remedies each give up something this proposal keeps. ASID and SPI-Active run a dedicated exploration episode on hardware and fit simulator parameters, and ASID then retrains the task policy in the calibrated simulator. This works but requires resets, a separate identification pipeline, and a parameter structure fixed in advance. BayRn tunes the randomization distribution with Bayesian optimization over repeated real rollouts. Domain Contraction needs an externally supplied rough parameter estimate at runtime. RMA is closest in spirit, since it adapts zero-shot by predicting latent physics from recent joint and motor history. That signal exists during continuous locomotion but not during a ballistic strike or a single slide, where the informative event is the outcome after the action has finished. Domain Contraction makes exactly this point about open-loop primitives.
Three papers constrain the design. Pinto et al.'s Asymmetric Actor Critic combined a privileged-state critic, hindsight relabeling and visual randomization to solve sparse-reward picking and pushing zero-shot (5/5 on a real Fetch). Their critic saw geometric state, not physics parameters, and the tasks tolerated imprecision. A Theoretical Justification for Asymmetric Actor-Critic Algorithms proves, for linear approximators, that a full-state critic removes the aliasing error that arises when different hidden states look identical to the agent. Hidden physics parameters are such a hidden state, so this result is why the critic here conditions on them. The same paper warns that larger critic inputs can raise approximation error, which we test directly. Finally, Active Domain Randomization steers randomization using a discriminator on trajectory differences. That signal collapses when every early rollout fails in the same flat way, so we do not rely on trajectory-divergence signals for exploration.
Core Claim
Consider a task instance with randomized physics parameters $\xi$ and goal $g$. The robot gets up to $K$ attempts. Before attempt $k$ the policy sees the history $h_{k-1} = \{(a_j, o_j)\}_{j<k}$ of earlier action parameters $a_j$ and observed outcomes $o_j$, where an outcome is the final object pose after the attempt. It chooses $a_k \sim \pi(\cdot \mid g, h_{k-1})$. Training maximizes
$$ J(\pi) = \mathbb{E}_{\xi \sim p(\xi),\, g}\Big[\textstyle\sum_{k=1}^{K} \gamma^{k-1}\, \mathbb{1}[\|o_k - g\| < \epsilon] \textstyle\prod_{j<k}\mathbb{1}[\|o_j - g\| \ge \epsilon]\Big], $$
where $\gamma<1$ rewards succeeding early and $\epsilon$ is the task tolerance. The reward is sparse because only the first success counts. The critic $Q(g, h_{k-1}, a_k, \xi)$ sees the true $\xi$ during training only.
Hindsight relabeling fits this structure cleanly. The history is a record of absolute outcomes that does not depend on the goal, so any recorded $o_j$ can be relabeled as the goal of the whole episode without making the history inconsistent. The episode is then truncated at attempt $j$. Every failed simulated episode thus yields several successful training examples.
The claim has two parts. First, a policy trained this way learns to read the physics from the outcomes of its misses and correct for them. Second, it may learn to make its first attempt deliberately informative, obtaining through sparse task reward the kind of probing ASID obtains from Fisher-information curiosity. The second part is the more speculative one, and the experiments are designed to confirm or refute it separately from the first.
Research Program
The first step builds the multi-attempt simulation environments in robosuite (robosuite) on MuJoCo, using operational-space control, which robosuite reports makes exploration substantially easier than joint-velocity control. We will build three task families, each chosen so that an attempt is short, its outcome is informative, and retrying needs little or no human reset.
The first family is shuffleboard, modeled on ASID. The puck is slid toward a zone under randomized friction and mass, and a return mechanism or a second puck provides the retry. The second family is Hit, modeled on Domain Contraction. A YCB-like box is struck to a target pose on randomized surfaces. The third family is peg insertion with force-terminated motion primitives in the style of Learning Insertion Primitives with Discrete-Continuous Hybrid Action Space. A failed insertion leaves the peg near the hole, so the retry comes almost for free. Here we randomize friction, clearance and the compliance of the contact.
In all three tasks the action for one attempt is a low-dimensional parameter vector: strike velocity and direction, or primitive type plus its continuous parameters. The policy is therefore a recurrent or transformer model over a history of at most a handful of attempts. Keeping the per-attempt action small makes the learning problem manageable and is also what makes outcomes informative.
The second step is training. We use an off-policy actor-critic with a critic that conditions on the true parameters, together with the relabeling scheme above. Randomization ranges are set wide enough that single-attempt randomization is measurably conservative in simulation, which we confirm before any hardware work. The third step analyzes what the policy has learned, in two ways. We fit a linear probe from the policy's internal state after each attempt to $\xi$, and we measure how informative first attempts are, by computing the posterior contraction over $\xi$ that an attempt produces compared with the attempt a parameter-aware oracle would choose.
The fourth step is zero-shot hardware evaluation on a Franka Panda, with outcomes read from a pose tracker. Visual randomization is kept out of the core claim; if we add it, we follow Pinto et al.
Baselines
The naive reference point is a single-attempt policy trained with uniform randomization, the asymmetric critic and hindsight relabeling. This is Pinto et al.'s recipe applied to our tasks and allowed to retry, but its retries do not depend on history. It shows what randomization alone gives. The number to beat on shuffleboard is ASID's reported 30% for randomization, although our task instance will differ.
The strongest existing method is ASID: one exploration episode on hardware, a CEM fit of simulator parameters, and a task policy retrained in the calibrated simulator. ASID reports 70% success on shuffleboard and 66% (6/9) on rod balancing. For a fair comparison, ASID's exploration episode counts as one of our attempts. We compare ASID's success on its first task attempt against our success by attempt 2, with the same simulator, the same randomization ranges for its prior, and matched simulation compute including its retraining. Beating ASID means matching or exceeding its success at an equal count of hardware interactions with no retraining and no separate exploration phase. For Hit, the reference is Domain Contraction's reported 0.015 final-state error with a supplied parameter estimate. We will run Domain Contraction with the estimate produced by a CEM fit to our first attempt's outcome, which is its natural zero-shot variant.
The third baseline is the teacher-student approach in the style of RMA. A teacher conditioned on $\xi$ is trained with the same reward and relabeling, and a student is trained to regress the teacher's latent from attempt history. It sees exactly the same information as our policy, so it isolates whether learning end to end over attempts matters. We also include an oracle, a policy given the true $\xi$, as an upper bound, since the gap between the naive baseline and this oracle measures conservatism.
Two ablations isolate the key ingredients. The first uses a history-conditioned critic that does not see $\xi$ and tests the aliasing argument. The second removes hindsight relabeling and tests whether the sparse multi-attempt objective is learnable without it. All methods receive the same simulation steps and the same hyperparameter search budget, with five seeds in simulation.
Why not just…?
Why not just use uniform randomization with an asymmetric critic and hindsight relabeling, as Pinto et al. did? Their tasks were coarse picking and pushing, where hedging costs little. On precision tasks, randomization-trained policies fail badly, with 0/9 on rod balancing in ASID and 0.653 error on Hit in Domain Contraction. Our naive baseline is exactly this recipe, so the experiments measure its shortfall directly and do not assume it.
Why not just distill a parameter-aware teacher into a history-reading student, as RMA does? This is the real threat, because it sees the same history and is simpler to train. It may match us on correction after a random first miss. It cannot learn to probe, however, because the student imitates a teacher that already knows $\xi$ and so never needs to gather information. If the student matches our policy at every attempt count, the probing half of our claim is wrong, and we will report the simpler method as sufficient. That comparison is the decisive one.
Why not just shape a dense reward in simulation? Distance-to-target terms are easy to write for shuffleboard. But a dense reward does not change what the policy can observe, and a policy that cannot identify the physics still hedges no matter how it was rewarded. We will include a shaped-reward single-attempt baseline to show that its limit is observability, not exploration.
Why not just calibrate with ASID? It works, and we compare against its reported numbers. The cost is a separate exploration phase with resets, a parameter model fixed in advance, and retraining for every new setup. Our policy instead adapts across attempts at deployment and needs no retraining.
Proposed Experiments
The first decisive check is a curve of cumulative success against attempt number, for $K$ from 1 to 5. The proposal predicts a crossover. The adaptive policy may match or trail the naive baseline on attempt 1, if it probes, and then clearly exceed it by attempts 2 to 3, approaching the oracle. A policy that is merely better across the board on attempt 1 would suggest it learned something other than adaptation.
The second check addresses whether first attempts are chosen to be informative. We compare the posterior contraction over $\xi$ produced by our policy's first attempt against that of the naive baseline's first attempt, and against the teacher-student model.
The third check covers the ablations. We expect the ablation without relabeling to fail to learn under the wide randomization ranges, and the ablation whose critic lacks $\xi$ to learn more slowly and settle on more conservative policies. The Theoretical Justification paper predicts the second effect under linear approximation, and this measures whether it holds for deep networks.
The fourth check is hardware transfer. We run 20 trials per task on the Franka, reporting success by attempt $k$ for shuffleboard and insertion, and final-state error for Hit. The same instances are run by the ASID-style and Domain Contraction baselines as described above.
Finally, a stress test shifts the real parameters outside the training range, using a heavier puck or a waxed surface. It checks whether history-based correction degrades gracefully where explicit identification with a fixed parameter model would stall.
References
- MANGA: Method Agnostic Neural-policy Generalization and Adaptation
- Robust Manipulation Primitive Learning via Domain Contraction
- ASID: Active Exploration for System Identification in Robotic Manipulation
- Sampling-based System Identification with Active Exploration for Legged Sim2Real Learning
- Data-Efficient Domain Randomization With Bayesian Optimization
- RMA: Rapid Motor Adaptation for Legged Robots
- Asymmetric Actor Critic for Image-Based Robot Learning
- A Theoretical Justification for Asymmetric Actor-Critic Algorithms
- Active Domain Randomization
- robosuite: A Modular Simulation Framework and Benchmark for Robot Learning
- Learning Insertion Primitives with Discrete-Continuous Hybrid Action Space for Robotic Assembly Tasks