Asymmetric Actor-Critic Training with Privileged Information

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2253 papers366 conceptsfrozen 2026-10-04

In the controlled environment of a physics simulator, we are "gods." We know the exact friction coefficient of a floor, the precise mass of a manipulated object, and the hidden velocities of every joint. Yet, when we deploy a robot in the real world, it is "blind" by comparison, relying on noisy cameras, laggy sensors, and partial observations.

The research direction of Asymmetric Actor-Critic training bridges this gap. By allowing the "Critic" (the evaluator) to see the full, privileged state of the simulation while restricting the "Actor" (the controller) to only what it can see in reality, researchers have unlocked a powerful scaffold for sim-to-real transfer.

The Information Gap in Partial Observability

Most real-world robotic tasks are Partially Observable Markov Decision Processes (POMDPs). A quadrotor flying through a forest, as explored by Jonas Eschmann, Dario Albani, and Giuseppe Loianno in Learning to Fly in Seconds, does not know the exact wind disturbances or its ground-truth state; it only has noisy IMU and motor data.

If we train a standard reinforcement learning (RL) agent on these noisy signals, the learning process is incredibly unstable. The agent might take a perfect action but get a poor reward because of a hidden variable (like a sudden gust of wind). To a standard Critic, this looks like the action was bad, leading to high-variance gradients that stall learning.

The Asymmetric Solution

The core insight, popularized by Lerrel Pinto and Marcin Andrychowicz in Asymmetric Actor Critic for Image-Based Robot Learning, is to decouple the information available to the two halves of the RL algorithm:

  1. The Critic (The Teacher): During training in simulation, the Critic is given privileged information—ground-truth object coordinates, forces, and environmental parameters.
  2. The Actor (The Student): The Actor is restricted to observable information—raw pixels, proprioception, or tactile signals—that will actually be available on the physical robot.

Because the Critic knows the "true" state of the world, it can provide a much more accurate value estimate. It can distinguish between a "good move made in a difficult environment" and a "bad move made in an easy one." As Gaspard Lambrechts and colleagues argue in A Theoretical Justification for Asymmetric Actor-Critic Algorithms, this effectively eliminates state aliasing—where two physically different situations look identical to a sensor-limited robot—allowing for faster convergence and higher performance.

Building Intuition: The Scaffolding Effect

Why does giving the Critic more information help the Actor, even if the Actor never sees that data?

Think of it as a coach watching a player from the sidelines with a high-speed camera and thermal imaging. The player (the Actor) only sees what’s in front of them. The coach (the Critic) sees the whole field and the player's internal fatigue levels. The coach uses this "privileged" view to give the player a very precise grade on their performance. Over time, the player learns to adjust their movements based on that high-quality feedback, eventually developing a "feel" for the game that implicitly accounts for the factors they couldn't see.

This mechanism is what allowed OpenAI to train a humanoid hand to Solve Rubik's Cube. The value network saw the noise-free state of the cube and hand, while the policy had to rely on noisy sensors. Similarly, Arthur Allshire and Mayank Mittal used this to Transfer Dexterous Manipulation to a real-world TriFinger robot, where the Critic accessed exact forces and velocities that the physical robot could never measure directly.

Applications Across Robotics

This asymmetric framing has become the standard "recipe" for modern sim-to-real transfer:

  • Legged Locomotion: In DTC: Deep Tracking Control, Fabian Jenelten and Marco Hutter use a privileged Critic to help quadrupeds navigate slippery or moving surfaces. By seeing the ground-truth friction during training, the Critic helps the Actor learn a policy that is robust to those variations in the real world.
  • Blind Manipulation: Zhao-Heng Yin and Yuzhe Qin demonstrated in Rotating without Seeing that a robot can rotate objects using only touch. The Critic was trained with object mass and friction, enabling the Actor to learn a policy that "senses" these properties through tactile history.
  • Agile Flight: Ismail Geles and Davide Scaramuzza showed in Demonstrating Agile Flight from Pixels that drones can race at 40 km/h by mapping pixels to control. The Critic's access to ground-truth states during training was essential for the Actor to master maneuvers that are otherwise impossible to learn from noisy visual data alone.

Beyond Simple Asymmetry: Teacher-Student and Guiders

While the basic asymmetric actor-critic is powerful, newer work like Guided Policy Optimization under Partial Observability by Guangming Xie and Zongqing Lu suggests co-training a "guider" and a "learner." This ensures the privileged "teacher" doesn't become so advanced that the "student" can no longer imitate its behavior—a common failure mode known as the "imitation gap."

By treating simulation data not just as a playground, but as a source of privileged supervision, this research direction has turned the "unrealism" of simulators into their greatest strength. The result is a generation of robots that can act with the confidence of a god, even while seeing through the limited eyes of a machine.

Papers in this version

Showing 60 of 253 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 186 papers
  2. Version 1 · 2026-10-04 · 253 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_asymmetric_actor_critic_1ac672a2,
  title = {asymmetric actor-critic},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/asymmetric-actor-critic/rdr_1ac672a2d7cc4e3e996cab4f0fc4f00b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_asymmetric_actor_critic_training_with_pr_1ac672a2_v1,
  title = {Asymmetric Actor-Critic Training with Privileged Information},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/asymmetric-actor-critic-training-with-privileged-information/rdr_1ac672a2d7cc4e3e996cab4f0fc4f00b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}