Simulated environments are a paradise for reinforcement learning (RL). In simulation, data is cheap, safety is guaranteed, and training can be accelerated across thousands of parallel instances. Frameworks like SURREAL, introduced by Linxi Fan et al. in SURREAL: Open-Source Reinforcement Learning Framework and Robot Manipulation Benchmark, demonstrate how distributed data generation can rapidly optimize complex manipulation policies.
Yet, when these optimized policies are deployed on physical hardware, they frequently fail. This discrepancy between virtual training grounds and the physical world is known as the reality gap. It stems from two primary mismatches:
- Visual Discrepancies: Simulators approximate lighting, textures, and shadows. Real-world environments contain sensor noise, varying illumination, and complex backgrounds that neural networks trained on clean synthetic data fail to interpret.
- Physical Discrepancies: Physics engines simplify contact dynamics, friction, and joint latencies. Unmodeled complexities—such as the non-smooth dynamics of soft materials explored by Noel M. Naughton et al. in Elastica: A Compliant Mechanics Environment for Soft Robotic Control—cause policies to exploit simulation artifacts that do not exist in reality.
To bridge this gap without relying on prohibitively expensive real-world data collection, researchers have developed mechanisms that fall into three core paradigms: Domain Randomization, Grounded Action Transformation, and Privileged Distillation.
Domain Randomization: Training for Variety
The intuition behind domain randomization is simple: if the network treats the real world as just another variation of the simulator, it will generalize zero-shot. Instead of building a single, perfectly accurate simulation, developers randomize environmental parameters—such as camera angles, surface textures, friction coefficients, and mass—during training.
Standard Training: [Perfect Sim] ──> Overfits ──> [Real World (Fails)]
Domain Randomization: [Sim Var A] \
[Sim Var B] ───> Generalizes ───> [Real World (Succeeds)]
[Sim Var C] /
In (CAD)$^2$RL: Real Single-Image Flight without a Single Real Image, Fereshteh Sadeghi et al. applied this to indoor drone flight. By forcing a deep RL policy to navigate highly randomized 3D CAD environments with extreme variations in rendering and geometry, the policy learned to focus on invariant structural features (like wall boundaries) rather than specific textures. As a result, the drone successfully navigated real-world hallways without ever seeing a real image during training.
Similarly, for complex physical systems like quadrotors, Inkyu Sa et al. demonstrated in Control of a Quadrotor with Reinforcement Learning that randomizing physical forces and latencies yields a policy robust enough to stabilize a physical drone even when thrown upside-down into the air.
Grounded Action Transformation: Correcting the Physics
Domain randomization makes a policy conservative by forcing it to perform well across a wide envelope of environments. However, for contact-rich tasks like dexterous manipulation, this conservatism can degrade performance. An alternative approach is to explicitly align, or "ground," the simulator's physics with reality.
Grounded Action Transformation (GAT) inserts a learned model between the policy and the simulator. When the policy selects an action, an action transformer modifies it before passing it to the simulator, ensuring that the simulator's next state matches what would actually happen in the real world.
[Policy] ──> (Action $a$) ──> [Action Transformer] ──> (Modified Action $a'$) ──> [Simulator]
▲
Learned from Real-World Data
Traditional GAT methods struggle when policies are parameterized by deep neural networks, as small changes in state transitions can destabilize learning. To resolve this, Haresh Karnan et al. introduced Reinforced Grounded Action Transformation for Sim-to-Real Transfer (RGAT). RGAT frames the optimization of the action transformer as a reinforcement learning problem itself, allowing it to smoothly ground complex neural network policies.
When contact physics are too discontinuous to optimize directly, researchers relax the environment itself. In Parameterized Quasi-Physical Simulators for Dexterous Manipulations Transfer, Xueyi Liu et al. proposed QuasiSim, which utilizes a curriculum of "quasi-physical" simulators. By softening rigid contact constraints early in training, the optimization landscape becomes smoother, allowing the policy to learn complex hand-object interactions before transitioning to realistic, high-fidelity dynamics.
Privileged Distillation: The Teacher-Student Framework
In simulation, we have access to "privileged" information that a real robot cannot observe directly, such as the exact 3D coordinates of an object, contact forces, or friction coefficients. We can leverage this information using a teacher-student training pipeline:
- Train a Teacher Policy: An RL agent is trained in simulation with full access to the privileged state space. Because it can "see" everything, it learns an optimal control strategy quickly.
- Distill into a Student Policy: A student policy is trained to replicate the teacher's actions, but it is restricted to using realistic sensor inputs, such as raw RGB pixels, depth maps, or tactile data.
Simulation Phase: [Privileged State (e.g., Exact Coordinates)] ──> [Teacher Policy]
│
(Distillation)
▼
Deployment Phase: [Sensor Inputs (e.g., RGB / Tactile)] ─────────> [Student Policy] ──> [Real Robot]
This strategy allows the student network to learn how to infer latent physical properties directly from history or sensory streams. For instance, Kartik Sachdev et al. utilized a divide-and-conquer distillation strategy in Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids. By combining automated system identification with keypoint-based rewards, they successfully transferred bimanual manipulation policies to noisy, low-cost humanoid hardware.
A similar distillation philosophy applies when migrating slow, computationally heavy optimization algorithms to real-time hardware. Cristina Pinneri developed APEX (Extracting Strong Policies for Robotics Tasks from Zero-Order Trajectory Optimizers), a framework that distills the behavior of slow trajectory optimizers into fast, reactive neural network policies, enabling real-time execution on physical platforms.
By combining these three pillars—broadening experience via domain randomization, correcting simulation errors via action transformation, and transferring privileged knowledge via distillation—modern robotic frameworks achieve robust, zero-shot transfer from virtual playgrounds to the physical world.