A robot that learns a task from a handful of demonstrations has only been told what to do along the paths the demonstrator took. Many different reward functions explain those demonstrations equally well. They agree on the demonstrated path and can disagree sharply everywhere else. When the policy trained in simulation meets the real robot, small physics mismatches (friction, latency, contact) push it slightly off that path. The error then grows at every step, because each mistake lands the robot in states it never saw. Our claim is that these two familiar problems are one problem. The reality gap decides where the robot drifts, and reward ambiguity decides how badly it behaves once there, because in drifted states the learned reward chooses among recovery actions more or less at random.
We propose to measure this coupling directly and to train against it. We learn an ensemble of rewards that all explain the demonstrations. We train them across randomized simulator physics, so that disagreement caused by the physics is removed and only genuine off-path ambiguity is left. We then train a maximum-entropy policy that maximizes a pessimistic quantile of return across that ensemble, under randomized dynamics. Maximum-entropy reinforcement learning is reinforcement learning that rewards the policy for staying random where randomness costs little. The resulting policy prefers recovery actions that every plausible reward endorses. The test is simple. The benefit should grow with task horizon and with the size of the dynamics gap. If a strong single-reward method given the same randomization matches it, the ambiguity story is wrong.
Core Claim
The file supports two established pieces that have not been put together. Bayesian Inverse Reinforcement Learning and the sample-complexity packing argument in A Lower Bound for the Sample Complexity of Inverse Reinforcement Learning show that finite demonstrations leave a whole set of rewards indistinguishable. Efficient Probabilistic Performance Bounds for Inverse Reinforcement Learning shows that point-estimate proxies fail in unseen states, and it bounds that failure with a Value-at-Risk over posterior samples. Separately, imitation and sim-to-real work treats compounding error as a coverage problem. That work fixes it with hierarchy (BUDS, ManipGen), with resets to expert states (MMDP-SR), or with Bellman constraints that carry value from expert states outward (Dual Q-DM). All of these assume that train and test dynamics match, or that the reward is known.
We make one testable claim and one method claim. The diagnostic claim is this: along trajectories executed under a dynamics gap, the disagreement among demonstration-consistent rewards about which action to take predicts the step of first failure. It should do so better than policy entropy or state novelty alone. The method claim is this: training the policy against a low quantile of that ensemble, while the simulator physics are randomized, reduces compounding failure under the gap. It should do so more than either ingredient alone, and the margin should widen with horizon.
Research Program
Reward ensemble. The reward ensemble builds on the disentangling structure of Adversarial Inverse Reinforcement Learning (AIRL). AIRL splits its discriminator into a state-only reward $g(s)$ plus a shaping term $h(s') - h(s)$, and shows that $g$ transfers across changed physics (Disabled Ant, Shifting Maze). AIRL tested transfer between two fixed dynamics and kept a single reward. We make three changes. First, the discriminator gets $K$ reward heads $g_1,\dots,g_K$ that share a trunk. Second, each head is trained on a bootstrap resample of the demonstrations against one shared policy, which gives a cheap stand-in for posterior sampling. This avoids the per-sample MDP solve that makes MCMC in Bayesian inverse reinforcement learning impractical in continuous control. Third, training transitions are drawn from simulators with randomized physics parameters $\xi$. Because the shaping term absorbs physics-dependent structure, whatever disagreement remains among the $g_k$ reflects what the demonstrations fail to pin down, not what the simulator got wrong.
Policy objective. The policy is trained with soft actor-critic–style maximum-entropy updates on the following objective:
$$\max_\pi \; Q_{q}\Big[\, \mathbb{E}_{\xi\sim p(\xi)}\,\mathbb{E}_{\pi,\xi}\sum_t \gamma^t \big(g_k(s_t) + \alpha\,\mathcal{H}(\pi(\cdot\mid s_t))\big) \Big]_{k=1}^{K}$$
Here $Q_q[\cdot]$ is the $q$-th quantile over the $K$ reward heads, $p(\xi)$ is the physics randomization distribution, $\gamma$ is the discount, and $\alpha$ is the entropy temperature. With $q=0.5$ this reduces to a median-reward policy. As $q$ goes to 0 the policy hedges toward actions every plausible reward accepts. The entropy term plays the role described in Reinforcement Learning with Deep Energy-Based Policies, which found that soft policies keep multiple solution modes (100% vs. DDPG's 60% on the quadruped maze). Keeping several modes gives the quantile a set of candidate recovery actions to choose among, rather than a single collapsed one. For long-horizon manipulation, the ensemble and objective are applied to the low-level skills of a BUDS-style hierarchy. The known weak point of that hierarchy is that its skills are flat behavior cloning (BUDS).
Measurement. The program also includes a measurement study, which is useful even if the method fails. We will log ensemble disagreement along every executed trajectory, under both surrogate-real and real dynamics. We will then relate that disagreement to time-to-failure, and compare it against two alternative predictors: policy entropy and a state-density novelty score.
Baselines
The naive reference is behavior cloning on the same demonstrations, trained with physics randomization in simulation. It sets the floor that compounding error produces. RialTo reports that pure imitation drops to as low as 5% success under disturbances and distractors.
The strongest competitor on the compounding-error question is Dual Q-DM from Non-Adversarial Imitation Learning Provably Free of Compounding Errors. It carries value from expert states into unvisited ones, with linear-in-horizon error, and it avoids adversarial training. We run it with the same physics randomization $p(\xi)$, the same demonstrations, the same environment-step budget, and the same number of tuning trials for its Bellman penalty $\beta$ that we spend on $q$ and $\alpha$. There is no reported result found on Lacuna for Dual Q-DM as a scalar benchmark number. We therefore reproduce it on all our tasks. Beating it means a statistically reliable success-rate gain under the gap that grows with horizon.
The strongest competitor on reward transfer is single-reward AIRL with the same randomization. On Disabled Ant, AIRL's state-only reward scores 130.3 against GAIL's −58.8 (AIRL). We must at least match 130.3 in that setting and exceed AIRL on the longer-horizon tasks.
On long-horizon robosuite tasks, the reference is BUDS at an average of 66% on Tool-Use and Kitchen and 56% on the real Franka Real-Kitchen task (BUDS). We will reuse its hierarchy, so any improvement comes from the low-level objective alone.
For the dynamics-grounding angle, we include RGAT (Reinforced Grounded Action Transformation) combined with our learned median reward. RGAT reports near-optimal transfer to modified simulators but no scalar returns in the file. We therefore rerun it.
Three ablations isolate the key ingredient. The mean-over-ensemble objective ($q=0.5$) tests whether pessimism matters or only averaging does. The ensemble trained without physics randomization tests whether removing physics-entangled disagreement matters. A single reward with a tuned higher temperature $\alpha$ tests whether entropy alone produces the recovery.
Why not just…?
Why not just write a sparse task reward and run PPO with domain randomization, as RialTo and ManipGen do? This is the most serious practical alternative, and on tasks with an easy success check it may win. The file shows its limits in three places. RialTo reports that reinforcement learning from scratch exploits simulator bugs unless it is anchored to demonstrations. SURREAL left Bin Picking and Nut-and-peg Assembly unsolved even with 64 actors. Heavy randomization also yields conservative policies on contact-rich tasks. We will include a hand-rewarded PPO+randomization arm on the tasks where a reward can be written. If it matches us there, our contribution narrows to tasks specified only by demonstration, and we will say so.
Why not just run Dual Q-DM or AIRL with domain randomization? This is the direct threat to the claim, and it is why both are primary baselines at matched compute. Dual Q-DM's guarantees assume the transition operator is fixed. Under a train/test gap, the value it carries outward is computed under the wrong physics. Single-reward AIRL fixes one point in the ambiguous set and so ranks recovery actions arbitrarily off-path. If either matches the quantile policy as horizon and gap grow, then off-path ambiguity is not the bottleneck, and we will report that negative result.
Why not just increase the entropy bonus? Higher entropy widens coverage, as soft Q-learning shows. It does not tell the policy which of the covered actions the demonstrator would have wanted. The tuned-$\alpha$ single-reward ablation, given the same tuning budget as $q$, settles this question.
Why not just collect more demonstrations for behavior cloning? More data shrinks the ambiguous region but cannot cover drift states produced by physics the demonstrator never faced. The product rule cited by Instruction Agent (95% per step gives about 36% at 20 steps) means small per-step gains matter a great deal. We will sweep the number of demonstrations to show where, if anywhere, the gap to behavior cloning closes.
Proposed Experiments
The experiments run in three stages.
Stage 1: controlled gaps in simulation. We start with MuJoCo Disabled Ant, a direct comparison to AIRL. We add RGAT-style modified InvertedPendulum, Hopper and HalfCheetah as surrogate-real targets. Finally, we build a robosuite long-horizon task (BUDS Kitchen) whose "real" copy has shifted friction, mass and action latency. We sweep gap magnitude and task horizon (by chaining stages), and report success rate, return, and step of first failure.
Stage 2: harder sequential tasks. We add RLBench's longer tasks, where ChainedDiffuser reports 80.9% on continuous interaction tasks (ChainedDiffuser). Here we train only in a randomized RLBench and evaluate under a held-out physics setting, to check behavior at horizons up to hundreds of steps.
Stage 3: real hardware. We deploy on a real Franka Panda, repeating the Real-Kitchen protocol with 15–50 demonstrations.
Decisive checks. Two results determine whether the idea holds. The first is whether ensemble disagreement on drifted states predicts failure. We will score this by the area under the ROC curve for predicting failure within the next $N$ steps. It must beat entropy and novelty predictors, and it must hold in surrogate-real and real rollouts, not only in simulation. The second is whether the low-quantile policy's advantage over Dual Q-DM+randomization and AIRL+randomization increases with both horizon and gap size, while the three ablations each recover only part of it. If disagreement predicts failure but the quantile objective does not help, the diagnostic stands on its own as a trigger for targeted resets or human queries. If neither holds, the coupling we propose does not exist at practical scales.
References
- Bayesian Inverse Reinforcement Learning
- A Lower Bound for the Sample Complexity of Inverse Reinforcement Learning
- Efficient Probabilistic Performance Bounds for Inverse Reinforcement Learning
- Bottom-Up Skill Discovery From Unsegmented Demonstrations for Long-Horizon Robot Manipulation
- Local Policies Enable Zero-Shot Long-Horizon Manipulation
- Efficient Inverse Reinforcement Learning without Compounding Errors
- Non-Adversarial Imitation Learning Provably Free of Compounding Errors: The Value Flow Mechanism
- Learning Robust Rewards with Adverserial Inverse Reinforcement Learning
- Reinforcement Learning with Deep Energy-Based Policies
- Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation
- Reinforced Grounded Action Transformation for Sim-to-Real Transfer
- SURREAL: Open-Source Reinforcement Learning Framework and Robot Manipulation Benchmark
- Instruction Agent: Enhancing Agent with Expert Demonstration
- ChainedDiffuser: Unifying Trajectory Diffusion and Keypose Prediction for Robotic Manipulation