Curricula That End at the Real Robot's Physics

Robots are usually trained in simulators and then deployed on hardware. Two separate literatures have each solved half of a problem that matters for that pipeline. Goal-oriented curricula, which choose which training task the agent practices next because it helps on a fixed target task, let agents learn sparse-reward behaviors that plain training never discovers. Their target is always a distribution someone wrote down by hand. Methods that identify simulator physics from small amounts of real data can say which friction, mass, and damping values are consistent with the actual robot. They then train the policy on that distribution all at once, with no curriculum. On multi-stage contact tasks such as stacking two blocks, training on a wide physics distribution all at once is exactly the regime where agents learn nothing.

We propose a self-paced curriculum over goals and physics together, whose endpoint is set by a few minutes of logged real-robot data rather than by a hand-written target. Early in training the curriculum is free to use physically "wrong" but easier settings, such as light objects, high friction, and nearby goals, as stepping stones. A penalty that grows over training then pulls the training distribution onto the set of physics values and goals the real data supports. That set is kept as weighted samples rather than a single Gaussian, so uncertainty that the data cannot resolve is preserved instead of averaged away. The claim to test is narrow: on multi-stage manipulation, an anchored curriculum reaches the real-data-supported physics with a working policy, and direct training on the identified distribution does not.

Core Claim

Self-Paced Deep RL (SPDL, with its probabilistic derivation in A Probabilistic Interpretation of Self-Paced Learning) chooses training tasks by trading off two things. It rewards the agent's expected return on the current task distribution, and it penalizes distance between that distribution and a target distribution $\mu(c)$ over task parameters $c$. The penalty weight grows over training, so the training distribution ends at the target. In that work $\mu$ is a known Gaussian.

On the identification side, DROPO fits the mean and variance of simulator physics by maximizing the likelihood of offline real trajectories. It reaches about 2cm pushing error on a Franka Panda from 13 seconds of data. DROPO then trains on that distribution directly.

We join the two by letting the real data define the target. Write the task parameters as $c=(g,\xi)$, where $g$ are goal parameters (object poses, targets) and $\xi$ are physics parameters. The target $\hat\mu_D$ is built from the real dataset $D$. It is the empirical distribution of logged real goals, multiplied by a set of physics samples weighted by their trajectory likelihood under $D$. The curriculum solves

$$\max_{\pi,\,p}\; \mathbb{E}_{c\sim p}\big[J(\pi,c)\big] \;-\; \alpha_t\, W_2\big(p,\ \hat\mu_D\big),$$

where $J(\pi,c)$ is the return of policy $\pi$ on task $c$, $p$ is the current training distribution, $W_2$ is the 2-Wasserstein (optimal-transport) distance, and $\alpha_t$ increases on a schedule.

We use an optimal-transport distance rather than SPDL's KL divergence for two reasons. It is well defined between a sample-based $p$ and a sample-based target. It also handles targets with several separate modes, which the file identifies as a known weakness of Gaussian SPDL.

The new ingredient is not "SPDL plus DROPO with a Gaussian target." It is treating physics as a curriculum variable whose final value is fixed by evidence. The curriculum may move through regions of physics that the real data rules out, as long as it arrives at regions the data supports.

Research Program

The work has three parts.

Building the target. We extend DROPO's likelihood objective from a single mean and variance to a weighted set of physics samples. In practice this means several CMA-ES restarts, followed by reweighting candidate parameter vectors by their transition likelihood. When the data cannot tell two parameter combinations apart, for example heavy-and-slippery versus light-and-sticky, both stay in the target. We also keep DROPO's idea of absorbing unmodeled effects as extra variance.

The curriculum itself. Training alternates two updates. The first is a policy update with PPO on samples from $p$, using an asymmetric critic that sees the privileged physics, as in TriFinger transfer. The second updates the sample set representing $p$. Each sample is moved along the gradient of estimated return, then pulled toward its optimal-transport match in $\hat\mu_D$ with strength $\alpha_t$. Return estimates for new $c$ come from a value function conditioned on $c$, as in SPDL.

Analysis. We will measure which physics settings the curriculum actually visits, and whether the agent forgets earlier skills as the distribution moves. Active Domain Randomization reports severe forgetting when the physics distribution shifts during training, and smooth interpolation is meant to prevent it.

The role each prior paper plays. CausalWorld motivates the work. It reports that PPO, SAC, and TD3 learn nothing after 100M steps under broad joint randomization, and that no baseline solves Stacking2. SimOpt constrains the design. It shows that over-wide randomization produces conservative, "lazy" policies, so the endpoint must be narrow where the data allows. DROPO and SPDL supply the methods we reuse.

Closest prior work. SS-ADR already co-evolves goals and physics, through asymmetric self-play and Stein variational particles. Its physics follow the agent's own competence boundary, not real data. Its goals come from a self-play partner, not from the real task distribution. It was tested on 3- and 4-joint arms. PAIRED and ADR likewise choose environments by regret or behavioral discrepancy, with no anchor to reality. Our method differs from all three in where the curriculum is required to end.

Baselines

Do nothing. The naive reference is PPO with the same architecture trained at nominal simulator physics. Next to it is uniform domain randomization over hand-set bounds, in the style of Peng et al.. That paper reports 0.89 ± 0.06 real Fetch pushing success with a recurrent policy, against 0.67 for a feedforward policy with randomization. On DROPO's Franka pushing task, uniform randomization gives about 10cm error.

Strongest existing method. This is DROPO's identified distribution used for direct training, which reports about 2cm on Franka box pushing. On that task, beating it means matching 2cm error with the same real data. On multi-stage tasks, beating it means a clear success-rate gain at equal simulation steps.

The obvious combination. SPDL with DROPO's Gaussian as $\mu$ tests whether our sample-based, transport-based target matters. The relevant SPDL figures are 9.35 ± 0.1 asymptotic reward on 3D Point Mass, against 2.34 for ALP-GMM, and 53.9 ± 0.4 on ball-catching, against 42.6 for GoalGAN.

Co-evolving baseline. We run SS-ADR with its physics ranges set to the identified support. No reported number for SS-ADR on our tasks was found on Lacuna.

Ablation. The ablation that isolates our key ingredient freezes physics at the target samples and lets the curriculum move only the goals. If this matches the full method, using physics as stepping stones adds nothing.

Fairness rules. Every method receives the same real dataset, the same policy network, the same simulation-step budget, and the same number of hyperparameter trials, with five or more seeds each.

Why not just…?

Why not just train directly on DROPO's identified distribution? This is the real threat. On single-stage pushing we expect parity, and we will report it honestly. The case for our method rests on multi-stage tasks. CausalWorld's results suggest that even a modest physics distribution, combined with goal variation, leaves Stacking2 unsolved without structure. If direct training on the identified distribution also solves Stacking2, the proposal fails, and the decisive experiment below is built to show this either way.

Why not pre-train at nominal physics and then fine-tune on the target? A two-stage schedule is a crude curriculum and may capture most of the benefit. We include it at equal compute. If it matches the full method, the continuous anchoring is unnecessary.

Why not SPDL with a Gaussian fitted to the data? When the real data is ambiguous, the posterior has several modes. A single Gaussian then either covers the gap between modes, which reproduces SimOpt's over-randomization, or drops one mode. We test this directly on a task built to have two indistinguishable explanations of the data.

Why not use online methods like BayRn or SimOpt? BayRn needed 700 real rollouts. SPM reaches 90% and 40% on its two tasks but requires interleaved real deployments. Our method needs only passive logs. We compare at a matched budget of real seconds.

Why not rely on GPU-scale simulation? TriFinger reaches 83% real success through scale. CausalWorld shows that 100M steps do not solve Stacking2. We run all methods with Isaac-scale parallelism, so scale is held constant across the comparison.

Proposed Experiments

Evaluation protocol. We use a "pseudo-real" setup in which the test simulator differs from the training simulator in ways the training parameterization cannot express. These include a shifted center of mass, an added actuation delay, and an altered contact model, following DROPO's unmodeled-asymmetry test. Ten to sixty seconds of trajectories are logged from the test simulator, and success is measured there.

Tasks. The tasks are CausalWorld Pushing, Pick-and-Place, and Stacking2 on TriFinger, and robosuite Pick-and-Place and Nut Assembly on the Panda. Pick-and-Place and Nut Assembly are reported as unsolved by standard baselines. We also use the REALM Franka simulator, whose task progression correlates with real performance at $r=0.921$, as a stronger proxy for reality. Its staged reach, grasp, lift, move scoring gives partial credit.

Hardware. We will replicate DROPO's Franka offset-center-of-mass pushing task, and add one two-stage pick-and-place task.

Metrics. We report pseudo-real or real success rate and final position error, simulation steps to a success threshold, seconds of real data used, and the spread of the training distribution over time.

Decisive checks. The first is Stacking2 and Nut Assembly under identified physics. Our method must beat direct DROPO training and the two-stage schedule by a clear margin across seeds. Otherwise the core claim fails. The second is the goal-only ablation, which tells us whether physics stepping stones contribute. The third is the bimodal-posterior task, which tells us whether the sample-based transport target beats the Gaussian. The fourth is single-stage pushing, where we expect only parity with DROPO's about 2cm. A large gain there would point to a bug or a confound rather than a win.