Robots trained in simulation are often taught by a "critic," a network that scores how good the current situation is. In the common asymmetric setup, the critic sees the simulator's ground-truth state while the policy sees only what the real robot's sensors will provide. The critic is discarded at deployment. This trick is behind real dexterous-hand and TriFinger results. It has a known weakness, though: the privileged state is a long, unstructured vector of poses, velocities, masses, frictions and colors, and most of it is irrelevant to the task. Those irrelevant but randomized variables make the critic's job harder and its training signal noisier. We propose to use the one thing a simulator offers that logged data never does, the ability to intervene. During training, we would perturb individual privileged variables, measure through short branched rollouts how much each perturbation changes the return under the current policy, and train the critic's input encoder to keep only the distinctions that change the outcome.
The core claim is that a critic encoder trained on these interventional comparisons reduces critic error and speeds up the policy in manipulation tasks with many randomized privileged variables. It should beat both full-state critics and critics whose inputs are chosen by observational statistics. Because the encoder is a function of state alone, it keeps the unbiasedness guarantees of asymmetric policy gradients. The policy never sees any privileged input. We expect no gain in small gridworld memory tasks, and we treat that as a prediction to confirm rather than a failure.
What Prior Work Establishes and Leaves Open
The motivation comes from three findings about asymmetric critics. A Theoretical Justification for Asymmetric Actor-Critic Algorithms analyzes a natural actor-critic with linear function approximation. It shows that asymmetry removes aliasing and inference error but raises approximation error, because the value of the full state is a more complex function than the value of the agent's history. Informed Asymmetric Actor-Critic proves that conditioning the critic on any state-dependent signal keeps policy gradients unbiased. It also finds that reward-irrelevant privileged features act as a "noisy TV" that degrades the critic. Its fix chooses a signal subset with a kernel independence test (HSIC) before training and a return-prediction gain afterward. Both criteria are observational and static, and they choose among given variables rather than learning a representation. Unbiased Asymmetric Actor-Critic shows that a critic conditioned on state alone is biased by history aliasing, which means that two different histories can lead to the same physical state but different futures. That paper shows the fix is to condition on both history $h$ and state $s$. We adopt the history-state form throughout and change only how $s$ enters.
One result constrains the design. Asymmetric Actor-Critic with Approximate Information State finds that adding predictive representation losses to an asymmetric critic gives parity at best and slows early learning on Shopping-5 and Cleaner. Its lesson is that a statistical auxiliary loss on top of a critic that already has the state adds nothing. Our objective is not auxiliary. It reshapes the critic's input using interventional information the critic cannot obtain from its TD targets. We also predict the same null result on the small tasks where that paper tested.
The method draws on causal representation learning, which tries to recover the abstract variables that actually drive outcomes from raw low-level data (Toward Causal Representation Learning). Reward-oriented Causal Representation Learning argues, in linear structural-equation settings, that an agent needs only a coarse causal representation sufficient for reward, recovered from interventions, and that full recovery is wasteful. We take that stance into nonlinear deep RL, applied to privileged simulator state rather than to observations. Causal Curiosity shows that agents can run self-chosen experiments in CausalWorld to isolate factors such as mass. We use simulator-level interventions instead of agent-level experiments, because privileged training exists precisely so that we can bypass the agent. CausalWorld supplies the programmatic intervention interface on masses, frictions, sizes and colors that makes this cheap.
Two near neighbors differ in kind. AACC compresses episode-level physical context for the critic with an encoder trained only by value gradients. It shows that feeding context to the actor hurts robustness, which we respect by never giving the actor privileged inputs. Invariant Causal Representation Learning for Generalization in Imitation and Reinforcement Learning learns invariant causal latents from observations across environments using an offline pipeline of an identifiable VAE followed by the PC causal-discovery algorithm. It has no privileged critic. Neither uses interventions on privileged state to train the critic's encoder.
Core Mechanism
The privileged state $s$ is split into named variables $X_1,\dots,X_d$. These include each object's pose and velocity, each mass and friction, link lengths, and visual attributes. Periodically during training, we save simulator states from the replay buffer. For a sampled state $s$ and variable $X_i$, we create an intervened copy $s' = \mathrm{do}(X_i = x')(s)$, with $x'$ drawn from the training randomization range. We then run $k$-step branched rollouts from both copies under the current policy, using shared action noise and finishing each with the critic's own bootstrap estimate. This gives an estimated return difference $\Delta_i(s)$. An encoder $f$ maps $s$ to a compact $z = f(s)$ and is trained so that distances in $z$ track these effects:
$$ \mathcal{L}_{\text{int}} = \mathbb{E}_{s,i,x'}\Big[\big(\lVert f(s) - f(s')\rVert - \Delta_i(s)/c\big)^2\Big], $$
Here $c$ is a running scale of return magnitudes. The critic $V(h, f(s))$ is trained by ordinary TD learning, and $\mathcal{L}_{\text{int}}$ is added with a fixed weight. The objective pulls together states that differ only in variables whose change does not alter the outcome, such as the color of a block, or the mass of a distractor object the policy never touches. It keeps apart states whose difference matters. Relevance is measured under the current policy, so the abstraction can change as the policy changes. A variable such as the second block's friction should become relevant once the agent starts stacking, which a filter fixed before training cannot capture.
Unbiasedness follows from the informed asymmetric result, since $f(s)$ is state-dependent and $h$ is retained. The encoder can only affect variance and approximation error. That is exactly the error the theoretical paper identifies, and it is the quantity our experiments measure directly.
Research Program
The first stage builds the machinery. We will implement branched interventional rollouts with state save and restore, both in CausalWorld and in an Isaac Gym TriFinger reposing task modeled on Transferring Dexterous Manipulation from GPU Simulation to a Remote Real-World TriFinger. That work ran 16,384 parallel environments on a single GPU, so a few percent of simulation spent on interventions is affordable. Every intervention step counts against the method's environment-step budget.
The second stage is a controlled "noisy TV" study. We will add $m$ randomized, reward-irrelevant privileged variables: extra distractor objects, colors and textures, and masses of untouched bodies. We will vary $m$ from zero to many and measure critic value error against Monte-Carlo returns, the variance of the policy gradient, and the sample efficiency of the policy.
The third stage tests whether interventional relevance differs from observational relevance where it should. We will build curricula in which a distractor variable is correlated with a causally relevant one early in training, for example color with mass, and the correlation is broken at a scheduled point, as CausalWorld curricula permit. An observationally selected or end-to-end-gated critic can route value through the proxy and should show a spike in value error after the break. The interventional encoder should not, because it never treated the proxy as relevant.
Baselines
The naive reference point is symmetric PPO, with critic and actor sharing the same deployable observation history. This reproduces the setting in which CausalWorld reports that PPO, SAC and TD3 fail on Stacking2 and fail to learn at all under extreme randomization after 100 million timesteps. That paper gives no success-rate numbers for these failures.
The standard competitor is history-state asymmetric PPO with the full concatenated privileged vector, as in Unbiased Asymmetric Actor-Critic and the TriFinger work. It is the default in robot learning and is what our method must improve on. The only reported number on this task family is the 83% real-robot reposing success in the TriFinger transfer paper. No simulated reposing success for that critic was found on Lacuna, so we will re-measure it ourselves. For CausalWorld, no reported result was found on Lacuna for asymmetric critics at all.
The strongest existing method on this question is Informed Asymmetric Actor-Critic, whose critic receives an HSIC-selected subset of privileged variables. It directly targets the noisy-TV problem. No numerical returns were found on Lacuna for it; the file reports only that it beat all baselines on Car-Flag and gave "significant gains" on Memory-Four-Rooms. We will run its selection procedure on the same pre-training data budget our encoder uses.
For context on whether asymmetric critics help at all in locomotion, AACC reports a mean reward of 3271 on Laikago versus 2380 for domain randomization. We treat that as evidence that the asymmetric family is worth refining, not as a target on our tasks.
Three ablations isolate the key ingredient. The first is an end-to-end sparse-gated critic, with a learned per-variable gate under an L1 penalty trained only by TD error. It tests whether interventions add anything beyond learned selection. The second is an observational twin of our encoder, trained with the same metric loss but with $\Delta$ replaced by differences in predicted return between buffer neighbors. It tests whether intervening, rather than only observing, matters. The third is hard interventional selection, which masks variables by their average $\Delta_i$ without learning an encoder. It tests whether representation learning matters beyond variable selection.
All methods share network sizes, PPO hyperparameters tuned once on the full-state baseline, an equal search budget for each method's own hyperparameters, and identical total simulator steps including intervention rollouts. Each comparison uses ten seeds. Beating the strongest method means the following three things. First, statistically separated area under the success curve against IAAC and full-state AAC at $m$ large. Second, lower critic value error at matched steps. Third, a smaller post-break error spike in the curriculum test.
Why not just…?
Why not just let the critic learn which inputs to ignore, with gating or a bigger network? This is the most serious threat. End-to-end gating costs no extra simulation and may suffice when distractors are independent noise. The approximation-error analysis in A Theoretical Justification for Asymmetric Actor-Critic Algorithms suggests extra capacity spent on irrelevant inputs is not free. The noisy-TV results in Informed Asymmetric Actor-Critic suggest TD error alone does not reliably prune them. Neither result is decisive for deep critics, however. The gated and enlarged-critic baselines at matched compute settle it. If they match our method across the distractor sweep and the curriculum break, we will report that interventions are unnecessary.
Why not just use observational signal selection, as IAAC does? HSIC measures dependence in the training distribution, so it keeps proxies that are correlated with relevant variables. It is also computed before training, so it cannot follow relevance that emerges as the policy improves. The correlation-break curriculum and the observational-twin ablation test exactly these two differences. If neither shows a gap, the interventional component is not earning its cost.
Why not just add a representation loss such as approximate information states or a reconstruction autoencoder? Asymmetric Actor-Critic with Approximate Information State found such losses redundant once the critic has the state, and harmful early in training. We include that baseline and expect it to match full-state AAC. If our method also shows no gain on the classic POMDP tasks, that matches our prediction that the benefit appears only when the privileged state is large and mostly irrelevant.
Why not just randomize everything, or distill a privileged teacher into a student? Neither addresses the critic's problem. Extreme randomization is exactly what fails in CausalWorld. On Laikago, AACC reports that the two-stage method RMA failed to reach positive reward. We keep domain randomization as the training distribution for every method rather than as a competitor.
Proposed Experiments and Decisive Checks
The main benchmarks are three CausalWorld tasks, Pushing, Picking and Stacking2, under increasing randomization and added distractors, and Isaac Gym TriFinger reposing with keypoint observations for the actor. The classic POMDP suite of Heaven-Hell, Shopping, Cleaner and Car-Flag serves as a check that the method preserves the unbiased history-state behavior and adds no benefit where none is expected. The metrics are the following. For success, we use volumetric overlap for CausalWorld and reposing success for TriFinger. We also measure critic error against Monte-Carlo returns and the variance of the policy gradient estimate. Finally, we track wall-clock and step cost including interventions.
There are three decisive checks. First, the gap between our method and full-state AAC should grow with the number of distractor variables. A flat gap would refute the mechanism. Second, against the gated and observational ablations, the gap should be clearest in the correlation-break curriculum. Third, on Stacking2, where all baseline model-free agents in CausalWorld failed, we will report whether any asymmetric variant reaches non-trivial overlap. An improvement in critic error without an improvement in policy success would show that variance in the critic is not the bottleneck for that task.
References
- A Theoretical Justification for Asymmetric Actor-Critic Algorithms
- Informed Asymmetric Actor-Critic: Leveraging Privileged Signals Beyond Full-State Access
- Unbiased Asymmetric Actor-Critic for Partially Observable Reinforcement Learning
- Asymmetric Actor-Critic with Approximate Information State
- Toward Causal Representation Learning
- Reward-oriented Causal Representation Learning
- Causal Curiosity: RL Agents Discovering Self-supervised Experiments for Causal Representation Learning
- CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning
- AACC: Asymmetric Actor-Critic in Contextual Reinforcement Learning
- Invariant Causal Representation Learning for Generalization in Imitation and Reinforcement Learning
- Transferring Dexterous Manipulation from GPU Simulation to a Remote Real-World TriFinger