Training a robot to navigate a forest or rotate a screwdriver in its palm is notoriously difficult because of the "reality gap." In simulation, we have perfect knowledge of the world; in the real world, sensors are noisy, lighting changes, and physical properties like friction are unknown.
The research direction of Privileged Information Distillation (often framed as a teacher-student architecture) has emerged as the dominant solution to this problem. Instead of forcing a robot to learn complex behaviors from scratch using messy sensor data, researchers first train an "expert" in a virtual world where it is allowed to "cheat." This expert's wisdom is then distilled into a practical controller that can function in the real world.
The Oracle and the Practitioner
The core intuition behind this approach is the decoupling of task logic from perception.
In the first stage, we train a Teacher Policy (the Oracle) using Reinforcement Learning (RL). This teacher has access to "privileged information"—data that is available in a simulator's backend but impossible to measure perfectly in reality. As noted by Joonho Lee and Marco Hutter in Learning robust perceptive locomotion for quadrupedal robots in the wild, this might include the exact friction coefficient of the ground, the precise mass of an object, or the hidden geometry of an obstacle. Because the teacher can "see" the underlying physics, it learns the optimal motor commands very quickly.
In the second stage, we introduce the Student Policy (the Practitioner). The student does not have access to the "cheats." It only sees what a real robot sees: noisy depth images, joint positions, and tactile feedback. The student’s goal is not to maximize a reward directly, but to mimic the teacher’s actions. By observing the teacher, the student learns to infer the hidden physical properties of the world from its own limited sensors.
Why Distillation Works Better Than Direct Learning
Directly training a vision-based robot with RL is often sample-inefficient and unstable. If a robot falls over, is it because its legs moved wrong, or because the camera was slightly blurry?
Distillation simplifies the credit assignment problem. As Deepak Pathak demonstrates in Extreme Parkour with Legged Robots, the teacher provides a "dense" supervision signal. The student isn't just told "you failed"; it is shown exactly what the optimal movement would have been at every millisecond.
This framework allows robots to master feats that were previously impossible:
- Acrobatics: Dohyeong Kim and Songhwai Oh used this to teach quadrupeds backflips by breaking the motion into stages and distilling the expert's timing into a robust student model in Stage-Wise Reward Shaping for Acrobatic Robots.
- Dexterity: In A System for General In-Hand Object Re-Orientation, Tao Chen and Pulkit Agrawal showed that a student can learn to rotate complex objects by distilling a teacher that knows the object's exact center of mass and shape.
Learning to "Feel" the Environment
One of the most powerful aspects of this direction is implicit estimation. When a student mimics a teacher that knows the ground friction, the student eventually learns that certain patterns of "foot slip" in its sensors imply "low friction."
This is evident in the work of Takahiro Miki, where a robot learns to switch between "sprinting" when its vision is clear and "feeling" its way forward when its cameras are obscured by snow or fog. The student policy, through distillation, develops a form of "robot synesthesia," as explored by Ying Yuan in Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensing, where vision and touch are fused to compensate for each other's weaknesses.
Scaling to General Agency
The field is now moving toward using this distillation process to create more general controllers. Instead of training a new teacher for every task, researchers like Yixin Lin are exploring how foundation models can act as high-level teachers. In Proc4Gem, privileged simulation data is used to fine-tune large multimodal models, bridging the gap between high-level reasoning and low-level contact physics.
By treating simulation as a source of "perfect knowledge" and distillation as the bridge to "imperfect reality," this research direction has turned the sim-to-real gap from a barrier into a training feature. The result is a generation of robots that are not just programmed to move, but have learned to perceive and adapt to the complexities of the physical world.