Do Privileged Critics Make Domain-Randomized Policies Too Timid?

Robots trained in simulation are usually made robust by domain randomization: the simulator's physics, such as friction, mass, and motor delay, is resampled every episode, so the policy cannot rely on any single setting. The robot's controller never sees these parameters. To the controller they are hidden state, and the problem is partially observable. The other half of the network, the critic that scores actions during training, is usually handed the true simulator state and the sampled parameters directly. Theory from partially observable reinforcement learning says this shortcut is biased. A critic that sees the true parameters scores every situation as if the uncertainty were already resolved. It therefore cannot give credit to actions whose only purpose is to resolve it, such as a test tap to feel friction or a small nudge to find an object's center of mass.

We propose to test a specific explanation for a well-known complaint about domain randomization, that it produces conservative, "average" policies. Part of that conservatism may not be inherent to randomization. It may come from the biased critic, which makes probing look worthless. If so, conditioning the critic on the actor's interaction history as well as the privileged state and parameters should restore probing behavior and recover precision. This requires no explicit system-identification module, no extra reward, and no real-world adaptation loop. The question can be settled by measurement, and the answer matters either way. If the bias is negligible in practice, current sim-to-real practice is vindicated. If it is large, a one-line architectural change addresses a problem that has spawned entire adaptive-randomization pipelines.

Core Claim

Let $s_t$ be the simulator state, $\theta$ the randomized physics parameters sampled for the episode, and $h_t$ the actor's history of observations and actions. The standard privileged critic learns $V(s_t,\theta)$. For a history-dependent policy $\pi$, this quantity is an average:

$$V^\pi(s,\theta) = \mathbb{E}_{h \sim P(h \mid s,\theta,\pi)}\big[V^\pi(h,s,\theta)\big].$$

The average runs over all the histories the actor might have at that moment. Two situations with the same physics, one where the actor has just probed and knows $\theta$ and one where it has not, receive the same value. The reward of acting on knowledge is smeared across both. As a result, the advantage of the probing action is pushed toward zero.

Unbiased Asymmetric Reinforcement Learning under Partial Observability and Unbiased Asymmetric Actor-Critic for Partially Observable Reinforcement Learning proved that the history-state critic $V(h,s)$ removes this bias. They also showed that state-only critics fail outright on Heaven-Hell, a navigation task whose solution requires a deliberate information-gathering detour. Both papers studied small discrete gridworlds without randomized physics.

Our claim is that randomized physics parameters are exactly this kind of hidden variable. The bias should therefore appear in continuous, contact-rich sim-to-real tasks, and its size should grow with the value of information: how much a probe changes the best downstream action, relative to what the probe costs. A critic of the form $V(h,s,\theta)$ should close the gap.

This is distinct from the closest existing work. Asymmetric Actor Critic for Image-Based Robot Learning introduced the privileged-critic recipe for sim-to-real, but used a state-only critic, randomized only visuals, and tested tasks where probing is unnecessary. The unbiased-critic papers never randomized dynamics. Adaptive approaches attack conservatism with separate machinery instead of fixing the learning signal:

Research Program

The first component is a task family with a controllable value-of-information knob, built in Isaac Gym so that thousands of randomized environments run in parallel. The core task is Franka pushing or sliding with randomized friction and mass. The robot must place a block precisely, and the correct push depends on $\theta$. A short, slightly costly "test push" reveals $\theta$ through the resulting motion.

A second task reconstructs the scooping setup from CAPTURE. An asymmetric object has a randomized center of mass, and a small lift-and-tilt probe reveals it. In both tasks we vary two dials independently: the probe cost, and the spread of the randomization range. Together they set how much an informed policy beats a passive robust one.

We add one control task where probing has no value, DM Control walker with randomized link masses. There the bias should be near zero, which keeps the claim falsifiable.

The second component is the set of critics, compared under one actor. The actor is a gated recurrent unit (GRU) policy on proprioception and, in a later stage, on images. We choose a GRU because POPGym found it the most reliable memory model for reinforcement learning, while linear transformers and state-space models were unstable. The critics are:

  • $V(s)$
  • $V(s,\theta)$
  • $V(h)$, which is symmetric and unbiased but uses no privileged information
  • $V(h,s,\theta)$
  • $V(h, s, \theta_{\text{rel}})$, where $\theta_{\text{rel}}$ is the subset of parameters that is actually informative about reward

The last variant follows Informed Asymmetric Actor-Critic. That paper showed that feeding the critic irrelevant privileged signals injects structured noise, and that a selected subset can outperform the full state. Its finding constrains our design. Because randomization commonly samples dozens of parameters (30 for Humanoid in the ADR benchmark), the full-$\theta$ critic may lose to a selected one. We use the paper's informativeness test to choose $\theta_{\text{rel}}$.

A Theoretical Justification for Asymmetric Actor-Critic Algorithms adds a second constraint: a richer critic input raises approximation error when capacity is limited. We therefore sweep critic width.

The third component is a direct measurement of the bias, independent of final returns. In simulation we can roll the frozen policy out many times from matched $(s,\theta)$ pairs whose histories differ only in whether a probe has occurred. This gives Monte Carlo estimates of $V^\pi(h,s,\theta)$. From these we compute the advantage each critic assigns to the probe action and compare it with the true advantage. In the simplest one-parameter version of the push task, we also compute a near-Bayes-optimal probe/no-probe threshold by discretizing $\theta$. This tells us where an unbiased learner should start probing as the probe cost varies.

The prediction is concrete. The $V(s,\theta)$ critic's probe advantage should shrink toward zero, and its policies should stop probing at a lower value of information than the optimal threshold. The $V(h,s,\theta)$ critic should track that threshold.

Baselines

The naive reference point is uniform domain randomization with a recurrent actor and a symmetric history-only critic $V(h)$, trained with Proximal Policy Optimization (PPO). This is what a practitioner gets with no privileged information. BGN and Pinto et al. report that history-only learners are slow; Pinto et al. found a symmetric image baseline below 40% on Fetch Pick where the asymmetric method was near perfect. Beating it means higher success at equal environment steps.

The strongest existing competitor is the standard privileged critic $V(s,\theta)$ with the same recurrent actor. It is the de facto sim-to-real recipe and the direct descendant of Asymmetric Actor Critic for Image-Based Robot Learning. That paper reports 100% (5/5) real-world success on picking, pushing, and moving blocks, and near-perfect simulated Fetch Pick. Those tasks need no probing, so the numbers set a ceiling on non-probing tasks rather than a target on ours. No reported result was found on Lacuna for any critic variant on probing-dependent randomized-dynamics tasks.

All critic variants share the actor architecture, randomization distribution, environment-step budget, and seeds (ten per cell). Each gets the same hyperparameter search budget, and critic widths are matched where possible. Beating this baseline means three things on the high-value-of-information settings: significantly higher precision success, a probe rate matching the optimal threshold, and no loss on the walker control.

For the scooping task, the external reference is CAPTURE's reported "at least 70% success rate in sim-to-real transfer on object scooping across three different objects," reached within 3–4 adaptation iterations. Our method would count as competitive if it matched that within a single episode and without episodic resets.

Two ablations isolate the key ingredient. The first is $V(h,s,\theta)$ versus $V(s,\theta)$ with identical inputs apart from the history encoding. Here the history is fed through a stop-gradient copy of the actor's GRU state, so that representation sharing cannot explain the difference. The second is $V(h,s,\theta_{\text{rel}})$ versus $V(h,s,\theta)$, which separates bias removal from the noisy-signal effect.

Why not just…?

Why not just add an intrinsic reward for information gain? Leveraging Mutual Information for Asymmetric Learning under Partial Observability does this and transferred zero-shot to a UR5. This is a real threat on raw success, because a curiosity bonus will induce probing. However, the bonus changes the objective. It cannot tell a probe worth its cost from one that is not, so it should over-probe once the probe cost is high or the randomization range is narrow. We include this method as a baseline and plot probe rate against the optimal threshold across the probe-cost dial. If it matches the unbiased critic everywhere, our contribution shrinks to a cheaper route to the same behavior, and we will report that.

Why not just distill a privileged teacher into an observation-only student? A teacher that sees $\theta$ never needs to probe, so there is no probing behavior for the student to copy. To Distill or Decide? shows that distillation hits a ceiling when the student cannot recover the teacher's privileged information, and that reinforcement learning overtakes it as stochasticity grows. We run DAgger from a $\theta$-aware teacher. The prediction is that the student stays at the passive robust level in the high-value-of-information settings.

Why not just estimate $\theta$ explicitly and condition the policy on the estimate? This is the route of CAPTURE, UPESI, and EXI-Net. It works but adds a separate model and, in those papers, multi-episode adaptation or test-time optimization. It also does not by itself teach the policy to gather the data the estimator needs. That is the "vicious cycle" the ADR benchmark documents, where a poor initial policy yields uninformative data. We compare against an estimator-plus-conditioned-policy pipeline trained in the same simulator. The test is within-episode success. If the estimator pipeline matches us, the finding becomes "a fixed critic is a simpler equivalent."

Why not just use hindsight baselines? Posterior Value Functions and PGIF condition value estimates on future trajectory information to reduce variance. These methods address credit-assignment variance, not the averaging-over-knowledge bias described above, and they are complementary to it. We treat them as a possible add-on, not a competitor.

Proposed Experiments

The decisive check is the bias measurement on the push task, crossed with the probe-cost dial. The hypothesis survives only if two things hold. First, the $V(s,\theta)$ critic's estimated probe advantage must be measurably below its Monte Carlo value. Second, the gap must grow with value of information and vanish on the walker control. A flat, near-zero gap would mean belief contraction happens too fast for the bias to matter, which falsifies the claim.

The second check is behavioral: probe rate and precision success across the dial for all critics and baselines.

The third check is sim-to-real. We take the push and scooping policies zero-shot to a Franka or UR5 arm, with 20 trials per object and friction surface. We report success and probe frequency, and compare against the scooping figure from CAPTURE.

The final stage adds visual randomization to the actor's input, following Pinto et al. This tests whether the effect persists when the history must also absorb camera variation. It is also where we expect critic capacity, per the theoretical justification paper, to become the binding constraint.