Robots are often trained in simulation with domain randomization: every training episode gets slightly different physics (mass, friction, motor delay), so the real robot looks like one more variant. Training then has two sources of noise. One is the policy's own random actions. The other is the luck of which physics was drawn. Standard policy-gradient methods average both together. A trajectory looks "bad" because its world was icy, and the action that produced it gets blamed. The safety mechanism that keeps updates small, a trust region (a cap on how far the action distribution may move per update, usually measured by Kullback-Leibler divergence, a standard measure of how different two probability distributions are), is also averaged over all visited states.
Our claim is that this pooling has a concrete failure mode. In hard physics variants the robot falls early, so those variants contribute few states to the average. The pooled trust region therefore barely constrains how much behavior changes there, and the hard variants are often the ones closest to the real world. We propose to stratify both noise control and the speed limit by physics context. We sample several rollouts per drawn physics setting and compare them only to each other, which gives an unbiased, critic-free baseline. We then enforce the trust-region budget separately inside each physics stratum, not on the pooled average. We expect the main payoff in the lower tail of transfer performance: worst-decile return across held-out physics and zero-shot return on off-envelope target dynamics.
Core Claim
Let $\xi$ denote the randomized physics parameters of an episode and $r_t(\theta)=\pi_\theta(a_t\mid s_t)/\pi_{\text{old}}(a_t\mid s_t)$ the importance ratio between new and old policy. Ratio-Variance Regularized Policy Optimization (R²VPO) shows that the local geometry of KL and $f$-divergence trust regions is governed by the variance of $r_t$. It replaces PPO's clipping with an adaptive penalty on that variance, but it was tested only on fixed-physics DeepMind Control tasks. Under domain randomization, the law of total variance splits this quantity:
$$\mathrm{Var}(r)=\mathbb{E}_{\xi}\big[\mathrm{Var}(r\mid\xi)\big]+\mathrm{Var}_{\xi}\big(\mathbb{E}[r\mid\xi]\big).$$
The pooled expectation over $\xi$ is weighted by how many states each physics variant contributes. Short, failing episodes therefore carry little weight. The second term measures how unevenly an update reshapes behavior across physics regimes. A pooled penalty cannot tell this term apart from ordinary exploration.
We propose an objective that penalizes the ratio variance within each stratum, with every stratum weighted equally regardless of its episode length. It also uses a group-relative advantage computed only among the $G$ rollouts that share the same $\xi$:
$$L(\theta)=\frac{1}{K}\sum_{k=1}^{K}\Big(\hat{\mathbb{E}}_{\xi_k}\big[r\,\hat A_{\xi_k}\big]-\lambda_k\,\widehat{\mathrm{Var}}_{\xi_k}(r)\Big).$$
Here $K$ is the number of physics contexts sampled per batch. $\hat A_{\xi_k}$ is the return minus the mean return of the group sharing $\xi_k$. $\lambda_k$ is a per-stratum dual variable, adapted by the same Lagrangian rule R²VPO uses, so that each stratum meets a common variance budget.
The baseline stays unbiased because $\xi$ is drawn before any action. Subtracting a function of $\xi$ is therefore exactly the state-dependent-baseline argument. The group-relative construction is borrowed from Group Relative Policy Optimization (GRPO), whose normalization across a group of samples for the same query is analyzed in Reinforcement Learning with Verifiable Rewards. There it is applied to language model prompts. Here the "prompt" becomes the physics context.
Relation to Prior Work
Two lines of work motivate the proposal. CausalWorld reports that model-free learners fail to learn anything under extreme domain randomization even after 100 million timesteps. This is the regime where we claim between-physics variance swamps the learning signal. RRLS benchmarks domain randomization and robust methods under standardized parameter sets and documents large seed-to-seed variance for adversarial training. It proposes no estimator fix.
Two further papers supply the methods we reuse. R²VPO provides the ratio-variance penalty, which we stratify. The GRPO analysis provides the critic-free group baseline, which we move from prompts to physics contexts.
Three papers constrain the design.
- Need for memory. Understanding Domain Randomization for Sim-to-real Transfer proves that bounded transfer gaps require history-dependent policies and real dynamics inside the simulator's support. We therefore use recurrent policies, and we test explicitly outside the support.
- Privileged critics. A Theoretical Justification for Asymmetric Actor-Critic Algorithms proves that a critic given privileged simulator state removes aliasing error, but leaves its variance unanalyzed.
- Noisy privileged inputs. Informed Asymmetric Actor-Critic shows that uncurated privileged inputs inject structured noise into the critic.
Together these make a $\xi$-conditioned critic the natural competitor to our group baseline, not a settled answer.
The closest sim-to-real trust-region work is A Trust Region Approach for Few-Shot Sim-to-Real Reinforcement Learning (FOOD). FOOD keeps the policy's transition distribution close to a few real trajectories using an adversarial discriminator. We use no target data and no discriminator. Our trust region is about how evenly the policy changes across simulated physics, not about agreement with reality. Reinforced Grounded Action Transformation puts a KL constraint on a learned action-correction model rather than on the task policy, and is likewise orthogonal. Relaxed State-Adversarial Policy Optimization (RAPPO) handles robustness by perturbing states toward low return. It does not touch the estimator or the trust region.
Research Program
We will build the method on top of an R²VPO implementation with a recurrent (LSTM) policy. Rollout collection will be organized so that each batch draws $K$ physics contexts and $G$ parallel rollouts per context. The RRLS ModifiedParamsEnv wrappers (SuReLI/RRLS) make this a matter of seeding parallel environments with shared parameters.
The analysis has three parts.
- Measuring the decomposition. During training we log both terms of the variance decomposition, along with the per-stratum KL actually incurred by pooled methods. This tests whether hard, early-terminating strata really receive disproportionately large behavior shifts under a pooled constraint. If they do not, the core premise is false and we will report that.
- Gradient variance. We measure policy-gradient variance directly by repeated batch resampling at fixed parameters. We compare the pooled baseline, the $\xi$-conditioned learned critic, and the stratified group baseline.
- Choosing G. We vary $G$ to find the point where the cost of repeated rollouts per context outweighs the variance reduction. The repeated-rollout cost per context is the main expense of the method.
Baselines
Naive reference. The do-nothing reference is PPO with clipping, a pooled learned value function, and uniform domain randomization over the same parameter envelope. This is the de facto robotics recipe. It is run with the same recurrent architecture, the same total environment steps, and the same hyperparameter search budget, measured in number of configurations tried per method.
Strongest learned-baseline competitor. This is PPO with domain randomization and an asymmetric critic that receives the true $\xi$. It removes between-physics variance through learning rather than grouping. That makes it the most likely method to match our variance-reduction ingredient. It gets the same budget, and its critic receives exactly the $\xi$ vector our groups share.
Strongest regularization competitor. This is pooled R²VPO under domain randomization. It is the existing method our objective modifies, so beating it isolates stratification.
Robust-RL references. M2TD3 and domain randomization as implemented in RRLS are run with the RRLS protocol. For these, no reported result was found on Lacuna for numerical returns on the RRLS tasks. The file states only that M2TD3 is best in static worst-case settings and domain randomization best on average. RAPPO serves as a state-adversarial reference, but no reported result was found on Lacuna for its MuJoCo returns.
Few-shot reference on the FOOD suite. FOOD reports a median Normalized Average Return of 0.37 across 8 off-dynamics tasks, versus 0.04 for DARC (FOOD). FOOD uses target trajectories and our method does not. Approaching 0.37 zero-shot would therefore be a strong result. Falling short of it would not count against us.
Ablations. Two ablations separate the ingredients. The first keeps the stratified group baseline but uses the pooled ratio-variance penalty. The second keeps the pooled baseline but uses the per-stratum penalty.
What counts as beating the baselines. The proposal succeeds if the full method beats both the $\xi$-critic baseline and pooled R²VPO on 10th-percentile held-out return. The improvement must be outside the 95% bootstrap interval over 10 seeds, and mean return must not drop.
Why not just…?
Why not just run PPO with massive domain randomization and more samples? CausalWorld's 100-million-step failure under extreme randomization suggests that sample count alone does not rescue pooled estimators. The R²VPO paper also shows PPO flatlining on sparse BallInCup and collapsing late on CheetahRun even without randomization. Still, larger batches shrink both variance terms. We will therefore give the PPO baseline a doubled batch size at equal total steps. If it closes the gap, our variance argument reduces to a batch-size argument.
Why not just condition the critic on the physics parameters? This is the real threat to the baseline half of the proposal, since it is unbiased for the same reason ours is. The case against it is indirect. Informed Asymmetric Actor-Critic shows that privileged inputs can inflate critic variance. The asymmetric actor-critic theory leaves that variance unbounded and warns of higher approximation error. Our direct gradient-variance measurements and the first ablation settle this question. If the $\xi$-critic matches the group baseline, we will adopt it and rest the contribution solely on per-stratum trust regions. The second ablation tests that remaining claim independently.
Why not just oversample or upweight hard physics settings? Reweighting changes which domains the objective cares about. It does not stop a single update from rewriting behavior in a regime it rarely sees. That reweighting also biases the objective toward a worst-case target is the conservatism problem RAPPO was designed to avoid. We will include a variant that sets the sampling weights of the pooled method so that each stratum contributes equal state mass. If this matches per-stratum penalties, the mechanism is really state reweighting, and we will report it as such.
Proposed Experiments
Main suite. The main experiments use Hopper, Walker2d, HalfCheetah, and Ant from RRLS. We train under randomized mass and friction and evaluate three things:
- mean, 10th-percentile, and worst return over a held-out grid inside the training envelope;
- the same statistics on a grid extending 50% beyond it, which tests support mismatch;
- seed variance over 10 seeds.
Zero-shot off-dynamics transfer. We train on simulation-side environments of the FOOD suite (Broken Joint, Heavy and Friction Cheetah, Low Fidelity Minitaur) with domain randomization. We then report Normalized Average Return on the targets zero-shot. The broken-joint and motor-model targets lie partly outside any mass or friction envelope. They therefore probe whether even per-stratum stability helps when the real dynamics are unmodeled.
Stress test. We run CausalWorld Pushing under the benchmark's extreme randomization. The question is whether any learning occurs at all where pooled baselines are reported to learn nothing.
Decisive check. The prediction that would falsify the mechanism concerns the logged per-stratum KL. Under pooled methods it should be largest in short-episode strata. Our method should equalize it, and that equalization should account for the 10th-percentile gain. If per-stratum KL is already flat under pooling, or if equalizing it does not move tail return, the per-physics trust region is not doing what we claim.
References
- Ratio-Variance Regularized Policy Optimization
- Reinforcement Learning with Verifiable Rewards: GRPO's Loss, Dynamics, and Success Amplification
- CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning
- Rrls: Robust reinforcement learning suite
- Understanding Domain Randomization for Sim-to-real Transfer
- A Theoretical Justification for Asymmetric Actor-Critic Algorithms
- Informed Asymmetric Actor-Critic: Leveraging Privileged Signals Beyond Full-State Access
- A Conservative Approach for Few-Shot Transfer in Off-Dynamics Reinforcement Learning
- Reinforced Grounded Action Transformation for Sim-to-Real Transfer
- Revisiting Domain Randomization via Relaxed State-Adversarial Policy Optimization