Per-Domain Gains: Making Average-Reward Learning Sound Under Domain Randomization

Robots trained in simulation are usually made robust by domain randomization: every episode draws new physics, such as masses, friction and damping, so the policy cannot overfit to one simulator. Separately, a line of work argues that continuing control tasks like walking should be trained for the long-run reward per step rather than a discounted sum. This "average-reward" criterion removes the fragile discount factor. These two ideas break when combined naively. Average-reward critics estimate one scalar, the reward rate, and subtract it at every step. Under randomization, each physics setting has its own long-run reward rate, and a heavy robot simply earns less per step than a light one. Subtracting a single shared rate leaves a per-step error in every domain, and with no discounting that error accumulates instead of fading.

We propose to make the reward rate a learned function of the physics parameters, which we call a per-domain gain. We also fix the arbitrary offset of the value function separately for each domain rather than once globally. The resulting critic is well defined under randomization. It also gives an explicit estimate of how well the policy will perform, over the long run, in any physics setting. That estimate has two uses. A robustness objective can target the worst fraction of domains directly, giving a tunable dial between average-case and worst-case training. And comparing predicted with measured gains on held-out physics gives a quantitative test of whether competence in simulation predicts performance outside the training range.

Core Claim

Domain randomization with physics held fixed for a long rollout turns the training problem into what the average-reward literature calls a multichain process. The hidden parameter $\xi$ never changes within a rollout, so the optimal long-run reward rate depends on which $\xi$ was drawn. Classical relative value iteration assumes a single gain shared by all states. It does this by subtracting a reference value, as in Reinforcement Learning Algorithms for Average-Payoff Markovian Decision Processes and An Average-Reward Reinforcement Learning Algorithm for Computing Bias-Optimal Policies, or a minimum value in continuous spaces, as in Approximate Relative Value Learning for Average-reward Continuous State MDPs.

Deep average-reward methods inherit this assumption. RVI-SAC adapts Soft Actor-Critic with one slowly updated rate $\rho$, and Average-Reward Soft Actor-Critic does the same within an entropy-regularized formulation. Both were evaluated only on nominal MuJoCo, where a single gain is correct.

Our claim has three parts:

  1. A single gain is structurally wrong under randomization.
  2. The error should appear as value drift whose size grows with the gap between a domain's true rate and the shared estimate.
  3. Replacing the scalar with a gain head $\rho_\phi(\xi)$, and fixing the value offset per domain, removes this drift and supports a direct risk-sensitive objective over domains.

The modified critic target is

$$Q_\theta(s,a,\xi) \leftarrow r(s,a) - \rho_\phi(\xi) + \mathbb{E}\big[Q_\theta(s',a',\xi)\big],$$

where $Q_\theta$ is the relative action value, $s,a$ are state and action, and $\xi$ is the simulator's physics parameter vector. The parameter $\xi$ is available to the critic during training but never to the deployed actor.

With one scalar $\rho$, any domain whose true gain $\rho^\pi(\xi)$ differs from $\rho$ forces $Q$ to absorb a residual of $\rho^\pi(\xi)-\rho$ per step. Over a rollout of length $T$, the critic's values in that domain shift by roughly $T\,|\rho^\pi(\xi)-\rho|$. This linear-in-$T$ prediction follows from summing an uncancelled constant along an undiscounted chain, and it is the first thing we test.

Research Program

The method has three pieces, and the first two are the core.

Gain head. A small network $\rho_\phi(\xi)$ is trained on a slower time scale than the critic. We borrow the two-time-scale delayed update that RVI-SAC uses for its scalar rate, but make the update conditional on $\xi$.

Per-domain offset. The arbitrary offset that average-reward values carry is now a whole function $c(\xi)$, not one constant. A centering penalty fixes it: the mean of $Q_\theta(\cdot,\cdot,\xi)$ over replay samples from domain $\xi$ is pushed to zero. This generalizes RVI-SAC's reference function from a global average to a domain-conditional one. It is the deep, approximate analogue of the "quotient space" in which Model-Free Robust Average-Reward Reinforcement Learning with Sample Complexity Analysis runs its Robust Halpern Iteration, where values equal up to a constant are treated as the same. That paper motivates our setting by framing sim-to-real as transition uncertainty under average reward. It differs from ours in three ways: its analysis is tabular, it assumes a generative model that can reset to any state, and it optimizes the worst case over an uncertainty set rather than a distribution over domains.

Actor and risk objective. The actor sees only a short history of observations and actions, so it must infer the physics implicitly. This is the usual asymmetric arrangement for sim-to-real, in which the critic sees privileged information the actor does not. Because the gain head scores each domain, we can optimize a tail objective directly:

$$J_\alpha(\pi) = \mathrm{CVaR}_\alpha\big[\rho^\pi(\xi)\big],\quad \xi \sim P.$$

Here $P$ is the randomization distribution, and $\mathrm{CVaR}_\alpha$ is the mean gain over the worst $\alpha$ fraction of domains. Setting $\alpha=1$ recovers ordinary average-case randomization, and $\alpha\to 0$ approaches worst-case training. We implement this by weighting actor updates toward replay samples whose $\rho_\phi(\xi)$ falls in the current lower tail. We treat this as a heuristic gradient estimator, not a theorem.

The motivation comes from RRLS. That benchmark reports that worst-case methods like M2TD3 "often sacrificed performance in the nominal (standard) environment," while domain randomization was "a powerful baseline for average-case performance." The tail objective lets us trace the frontier between these two regimes with one algorithm.

Predictive-validity diagnostic. The surgical-training literature separates two properties of a simulator. A simulator with construct validity tracks skill differences inside the simulator. A simulator with predictive validity forecasts real performance. The randomized DaVinci trial in The development of tissue handling skills is sufficient and comparable after training in virtual reality or on a surgical robotic system found that virtual training reached comparable final force control but needed roughly twice the repetitions (5.45 vs. 2.48 on Ring Tower). We borrow this lens for agents. After training, we compare $\rho_\phi(\xi)$ with the measured long-run reward rate on physics settings held out from training, including settings outside the training range. Reporting rank correlation and calibration error turns "does simulated competence predict out-of-distribution competence" into a number the method itself produces.

Baselines

Naive reference points. The simplest reference is plain discounted SAC trained only on nominal physics, as reported in the RRLS protocol. It measures how much any randomization helps.

Strongest average-case competitor. The main competitor is discounted SAC (and TD3) with domain randomization over the same parameter distribution, which RRLS identifies as the strongest average-case baseline. To keep this baseline honest, its discount factor gets a per-environment sweep over {0.99, 0.995, 0.999}. This is a larger tuning budget than our method receives, because RVI-SAC reports that SAC is highly sensitive to the discount factor, with Swimmer needing 0.999 while that value fails elsewhere.

Strongest worst-case competitor. For worst-case performance we compare against M2TD3 and RARL as implemented in RRLS via SuReLI/RRLS.

Key ablation. The ablation that isolates our main ingredient is RVI-SAC with domain randomization and a single scalar gain. We run it in two forms: with $\xi$ withheld from the critic, and with $\xi$ appended to the critic's input while the gain stays scalar. The second form tests whether simply conditioning the value network is enough. A further ablation keeps the gain head but uses global rather than per-domain centering.

Fairness and success criteria. All methods use the RRLS protocol of 5 million environment steps and 10 seeds, the same network sizes and replay settings, and an equal hyperparameter-search budget, apart from the extra discount sweep for discounted baselines. No reported result was found on Lacuna for numerical RRLS returns of DR, M2TD3 or TD3; the paper gives only the qualitative ranking above. No reported result was found on Lacuna for numerical RVI-SAC or ASAC returns either. We will therefore rerun every baseline ourselves rather than quote figures.

Beating the baselines means the following. At $\alpha=1$, our method should be statistically indistinguishable from or better than tuned discounted SAC with randomization on average-case return over the uncertainty set. For some $\alpha<1$, it should also beat M2TD3's worst-case return while losing less nominal performance than M2TD3 does.

Why not just…?

Why not just use discounted SAC with domain randomization and a tuned discount factor? Discounting makes the multichain issue harmless, because each domain's values stay bounded whatever its gain. This is the most serious threat to the proposal. Our case against it is partly practical: discount sensitivity documented by RVI-SAC, and a discounted critic that gives no per-domain long-run rate to target or validate. But if a well-tuned discount matches us on average-case and worst-case return across all environments, the robustness claim fails. What would remain is only the diagnostic value of the gain head. The experiment is designed so this outcome can occur and be reported.

Why not just run RVI-SAC with domain randomization, or give its critic the physics parameters? This is the obvious combination of the two directions, and we expect it to fail in a specific way. With a scalar gain, even a critic that sees $\xi$ cannot represent a bounded solution in domains whose true rate differs from the shared $\rho$. The mismatch has to go somewhere, so it appears as drift that grows with rollout length. The drift diagnostic, which measures per-domain mean Q against $|\rho^\pi(\xi)-\bar\rho|$ and rollout length, tests this directly. If the drift does not appear, because truncation and resets hide it in practice, then the gain head's value reduces to the tail objective and the diagnostic, and we will say so.

Why not just use adversarial robust training such as M2TD3 or RARL? RRLS reports that these methods trade away nominal performance, learn slowly, and vary heavily across seeds on Humanoid Stand Up. They also optimize a single worst point and offer no dial toward the average case. The CVaR sweep compares the two approaches on the same nominal-versus-worst-case plane. If M2TD3 lies on or above our frontier, our objective adds nothing.

Why not just port Robust Halpern Iteration directly? Robust Halpern Iteration and anchoring methods like Accelerating Value Iteration with Anchoring carry guarantees that depend on tabular spaces and generative-model resets. Neither condition holds for continuous robots. We reuse their quotient-space idea but do not claim their guarantees.

Proposed Experiments

The main testbed is RRLS. We use HalfCheetah, which has no termination, together with Ant, Hopper and Walker2d, which have falls and use RVI-SAC's automatic reset-cost mechanism. Training ranges cover the 1D–3D mass, friction and damping uncertainty sets. Evaluation covers three conditions: nominal physics, a grid over the training set, and parameters extrapolated outside the training range, which serve as a proxy for an unmodeled reality gap. Metrics are nominal return, mean and lowest-10% return across the grid, and long-run reward per step over extended continuing rollouts.

The second testbed is CausalWorld. It reports that extreme randomization leaves PPO, SAC and TD3 unable to "learn anything at all, even after 100 million timesteps." On its Pushing and Picking tasks with mass and friction interventions, we ask whether per-domain gains let learning proceed under wider randomization than discounted SAC tolerates. The metric is volumetric overlap. CausalWorld reports no success percentages, so we will rerun its baselines ourselves.

Four checks decide the outcome:

  1. Drift law. In RVI-SAC with randomization and a scalar gain, per-domain value offsets should grow roughly linearly in rollout length times the gain gap, and should vanish with the gain head.
  2. Frontier. The $\alpha$ sweep should dominate the convex hull of the DR and M2TD3 points on the nominal-versus-worst-case plane.
  3. Predictive validity. Spearman correlation between predicted and measured per-domain gains on extrapolated physics should exceed the correlation obtained from the discounted critic's normalized values on the same domains.
  4. Kill criterion. If tuned discounted SAC with randomization matches us on the first two checks, we will report the method as a diagnostic tool rather than a better learner.

Validation on physical hardware, for example on the TriFinger platform underlying CausalWorld, is an optional extension. No result in this note depends on it.