When training robots in simulation, we enjoy an omniscient view of the world. The simulator knows the exact 6D pose of every object, the friction coefficient of every surface, and the perfect depth map without sensor noise or occlusions. This omniscient data is known as privileged information.
However, when deploying that same policy onto physical hardware, this omniscience vanishes. A real robot must rely on noisy, occluded, and partial observations from onboard sensors like RGB-D cameras, tactile sensors, or motor encoders. This discrepancy creates a profound information gap between simulation and reality.
Trying to train a policy directly from raw sensory inputs to complex actions via reinforcement learning (RL) often fails. The agent must simultaneously solve two incredibly difficult problems: representation learning (interpreting noisy pixels or point clouds) and policy optimization (discovering the correct motor commands). Privileged information provides a clean shortcut to decouple these challenges.
The Teacher-Student Paradigm
The most common strategy to exploit privileged information is a two-stage distillation pipeline. First, an "expert" or "teacher" policy is trained with full access to the privileged simulation states. Because the state space is clean and low-dimensional, the teacher quickly discovers optimal behaviors. Second, a "student" policy is trained to replicate the teacher's behavior, but it is strictly constrained to use only deployable, real-world observations.
[Privileged Simulation States] ---> ( Teacher Policy )
|
[Action/Value Targets]
|
v
[Raw, Noisy Sensor Inputs] ---> ( Student Policy )
For instance, Ulrich Viereck et al. developed Penalized Q Cloning (PQC) to clone the value function of a full-state motion planner onto a policy that only processes raw depth images for peg insertion. Similarly, in humanoid and bimanual manipulation, Lei Han et al. and Bohan Zhou et al. use teacher-student pipelines to distill "god-mode" simulation trajectories into vision-language-action models and point-cloud-driven policies, respectively. This paradigm works because it transforms a hard exploration problem (finding the policy) into a supervised learning problem (mapping sensors to known actions).
Beyond Two-Stage Distillation: Curriculums and Co-Training
While the two-stage approach is highly effective, transitioning abruptly from perfect knowledge to noisy sensors can cause performance degradation. To smooth this transition, researchers have introduced curriculum-based and simultaneous training methods.
One approach is state dropout, where privileged information is gradually phased out during training. Shivesh Khaitan et al. demonstrate this in autonomous driving by initially providing the agent with ground-truth future trajectories of surrounding vehicles and systematically dropping this data out, forcing the agent to rely on current observations. Similarly, Atil Iscen et al. utilize a multi-stage pipeline that provides privileged guidance to a quadruped and gradually removes it to foster agile locomotion skills.
Alternatively, the process can be unified into a single training phase. Dikai Liu et al. developed the Unified Locomotion Transformer (ULT), which uses causal masking to allow a quadruped robot to learn from privileged simulation variables and real-world sensor data simultaneously, avoiding the overhead of a separate distillation step.
A Taxonomy of Privileged Information
The nature of privileged information varies depending on the task domain:
- Geometric and Kinematic States: In dexterous manipulation, simulators provide exact joint states, 6D object poses, or signed distance fields (Dmytro Pavlichenko et al.; Zhenguo Sun et al.). In human-object interactions, this can include part-level coordinates represented as relative movement dynamics (Zekai Deng et al.).
- Physical Dynamics: For legged locomotion or multiphysics tasks, privileged data includes exact ground friction coefficients, terrain height maps, or the ground-truth positions of every particle in a fluid or deformable body (Álvaro Belmonte-Baeza et al.; Eliot Xing et al.; Chenjia Bai et al.).
- Task Metadata and Feedback: Some systems rely on perfect environmental metadata, internal object IDs, or immediate success indicators that are impossible to automate in the real world (Shu Yu et al.; Xinyi Mao et al.; Junhao Shi et al.).
Why It Works: The Optimization Intuition
Why does training a teacher on privileged information help the student learn to see?
Consider the optimization landscape of reinforcement learning. When an agent only receives high-dimensional, noisy sensory inputs (like an RGB image), the policy must map millions of pixels to a few continuous motor actions. The reward signal is often sparse, and the gradient updates are incredibly noisy because a small change in lighting or camera angle can drastically alter the input features without changing the underlying physical state.
Privileged information provides a low-dimensional, invariant representation of the environment. Training the teacher in this space creates a smooth, dense reward landscape, allowing the RL algorithm to easily find the optimal control manifold.
Once this manifold is established, training the student becomes a problem of representation learning rather than policy exploration. The student's neural network is tasked with finding a mapping from noisy pixels to the teacher's clean latent states or actions. By decoupling the search for what to do from the search for how to see, privileged information dependency transforms an intractable sim-to-real challenge into a structured, highly efficient learning pipeline.
Key Foundational Work
Learning Perceptual Concepts by Bootstrapping from Human Queries (2021) — contributed 2 concept elements to this direction.
Learning visual servo policies via planner cloning (2020) — contributed 1 concept elements to this direction.
Visual Navigation in Real-World Indoor Environments Using End-to-End Deep Reinforcement Learning (2021) — contributed 1 concept elements to this direction.