Asymmetric Actor Critic Learning

342 concepts181 papers8 proposals

When training a robot to operate in the real world, we face a fundamental information mismatch. A deployed policy must act based on partial, noisy, and often delayed observations—such as raw camera pixels, sparse tactile clicks, or onboard IMU readings. However, learning a stable control policy directly from these high-dimensional or incomplete streams is notoriously difficult.

In simulation, we have an unfair advantage: we know everything. We have access to "privileged information"—the exact 3D coordinates of every object, surface friction coefficients, joint velocities, and precise contact forces.

The naive approach forces the agent to learn using only what it can observe in the real world, discarding this rich simulation data. Asymmetric Actor-Critic training resolves this tension by breaking the information symmetry between the entity that evaluates actions and the entity that executes them.

The Mechanics of Asymmetry

In standard Actor-Critic algorithms, both the actor (the policy network) and the critic (the value network) receive identical observations. In an asymmetric architecture, these inputs are decoupled. The actor is strictly restricted to deployable observations, while the critic is fed a privileged state containing ground-truth simulation data.

During Training (Simulation):
Privileged State (Friction, Poses, Forces) ───> [ Critic ] ───> Low-Variance Value Estimates
                                                                     │
                                                                     ▼
Noisy Observations (Pixels, Proprioception) ──> [ Actor  ] ───> Policy Updates

During Deployment (Real World):
Noisy Observations (Pixels, Proprioception) ──> [ Actor  ] ───> Actions

Why does this structural split work so well? As Amit Sinha et al. demonstrate in Asymmetric Actor-Critic with Approximate Information State, the primary mathematical benefit of this asymmetric information flow is a massive reduction in policy gradient variance.

A critic attempting to evaluate a state from a single noisy image or partial observation must simultaneously solve two distinct problems: reconstructing the underlying state of the world and estimating the expected future rewards from that state. This dual requirement introduces severe noise and variance into the learning signal. By giving the critic direct access to the true underlying state, we offload the reconstruction problem entirely during training. The critic can provide clean, low-variance value targets, acting as an omniscient guide that shapes the actor's behavior.

Because the critic is discarded after training, the actor remains fully deployable on physical hardware without requiring any privileged data during inference.

Scaffolding Complex Physical Skills

This architecture has become a cornerstone of modern sim-to-real transfer, particularly for contact-rich and highly dynamic tasks where physical properties are difficult to estimate on the fly.

Dexterous Manipulation

In their seminal work on high-DoF humanoid hands, OpenAI et al. used an asymmetric actor-critic setup to feed the critic exact object velocities and hidden physical states, while the actor relied solely on physical sensors. This allowed the policy to discover complex human-like strategies like finger gaiting entirely from scratch.

Similarly, Arthur Allshire et al. leveraged this asymmetry in Transferring Dexterous Manipulation from GPU Simulation to a Remote Real-World TriFinger to pass exact forces and velocities to the critic, achieving robust 6-DoF manipulation.

When vision is entirely absent, Zhao-Heng Yin et al. showed in Rotating without Seeing: Towards In-hand Dexterity through Touch that feeding the value network privileged information like object mass and friction allows a policy to manipulate irregular objects using only blind tactile history.

Legged Locomotion

For quadrupeds navigating rough terrain, the critic can see the exact height map of the upcoming ground or the precise impact forces during a fall. Yuntao Ma et al. utilized this in Learning Arm-Assisted Fall Damage Reduction and Recovery for Legged Mobile Manipulators to train recovery behaviors from high-impact falls, noting that without these hidden states provided to the critic, the policy fails to converge on successful strategies.

Simon Chamorro et al. applied a similar strategy in Reinforcement Learning for Blind Stair Climbing with Legged and Wheeled-Legged Robots to enable robots to climb stairs blindly by giving the critic access to terrain heights, forcing the actor to implicitly infer environmental properties from internal sensor patterns.

Overcoming Sensory and Temporal Limitations

Privileged information isn't just about physical constants like mass and friction; it can also be used to resolve sensory latency, partial observability, and safety boundaries.

Signal Delay

In real-world systems, there is always a lag between perception and action execution due to sensor processing and communication overhead. Dongsheng Li addresses this in Addressing Signal Delay in Deep Reinforcement Learning by providing the critic with non-delayed state information during offline training. This allows the critic to act as an oracle, aligning value estimates with the true state of the environment at the exact moment an action is executed, even though the actor must operate under a sensory lag during inference.

Visual Control and Latent States

Training a policy directly from raw pixels often suffers from sample inefficiency. Lerrel Pinto et al. demonstrated in Asymmetric Actor-Critic for Image-Based Robot Learning that a critic with full state access accelerates image-based actor training significantly compared to symmetric baselines.

Taking this further, Alex X. Lee showed in Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model that providing the critic with privileged latent state samples allows pixel-based agents to approach the performance levels of agents trained entirely on ground-truth states.

Safety Constraints

In safety-critical applications, agents must respect constraints they might not be able to directly perceive. Dongchi Huang et al. introduced this concept to world models in PIGDreamer: Privileged Information Guided World Models for Safe Partially Observable Reinforcement Learning. Here, critics use privileged information to provide highly accurate safety evaluations during training, ensuring the policy is optimized against true risks that remain invisible to the agent's primary sensors during deployment.

The Intuition of the Scaffold

The asymmetric actor-critic paradigm works because it separates the difficulty of evaluation from the difficulty of execution.

By treating privileged simulation data as a temporary training scaffold, researchers can bypass the high-variance, non-stationary hurdles of partial observability. The critic absorbs the complexity of the global environment, leaving the actor with a single, clear objective: learning a robust mapping from its limited sensory reality to optimal actions.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 186 papers
  2. Version 1 · 2026-10-04 · 253 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_asymmetric_actor_critic_1ac672a2,
  title = {asymmetric actor-critic},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/asymmetric-actor-critic/rdr_1ac672a2d7cc4e3e996cab4f0fc4f00b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_asymmetric_actor_critic_1ac672a2_v2,
  title = {asymmetric actor-critic},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/asymmetric-actor-critic/rdr_1ac672a2d7cc4e3e996cab4f0fc4f00b/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake