Training robots in the real world is a logistical nightmare. Reinforcement learning algorithms often require millions of interactions to discover effective control strategies. If executed on physical hardware, this trial-and-error process results in broken joints, depleted batteries, and unsafe operating conditions.
Simulators offer a compelling alternative by providing safe, highly parallelizable, and accelerated environments. However, a policy trained to perfection in a virtual world almost always fails when deployed on a physical robot. This discrepancy is known as the reality gap. It stems from unmodeled physical dynamics, sensor noise, and visual differences between rendering engines and real-world cameras.
To bypass costly real-world fine-tuning, researchers focus on zero-shot sim-to-real transfer—structuring the simulation and the learning process so that the resulting policy can immediately handle the complexities of the physical world.
Forcing Invariance via Domain Randomization
The most intuitive way to bridge the reality gap is to accept that the simulator is wrong, but ensure that it is wrong in every possible direction. If a policy is trained in a single, static simulation, it will inevitably overfit to the exact parameters of that environment—such as a specific friction coefficient or a precise actuator latency.
Dynamics randomization solves this by stochastically varying physical parameters (e.g., mass, link lengths, surface friction, and motor strength) across training episodes. As demonstrated in Sim-to-Real: Learning Agile Locomotion For Quadruped Robots, forcing an agent to optimize its reward across a wide distribution of physical environments prevents it from relying on brittle, simulator-specific quirks. The policy is forced to discover a robust, generalized control strategy that can withstand real-world physical perturbations.
A parallel challenge exists in visual perception. A neural network trained on clean, simulated images will fail when confronted with real-world shadows, lens flares, or background clutter. Visual domain randomization applies the same philosophy to rendering. Sadeghi et al. demonstrate in (CAD)$^2$RL: Real Single-Image Flight without a Single Real Image that maximizing the diversity of visual features—such as textures, lighting angles, and camera positions—in highly randomized, low-fidelity environments is far more effective for generalization than chasing perfect photorealism. Because the network never sees the same visual configuration twice, it learns to ignore the superficial "noise" of textures and lighting, focusing instead on the underlying spatial geometry required for tasks like collision-free flight.
High-Fidelity Physics as an Alternative to Randomization
While domain randomization builds robustness by exposing the agent to variation, an alternative school of thought argues that the reality gap can be minimized by eliminating modeling errors entirely. If a simulator can replicate physical interactions with near-perfect fidelity, a policy can transfer zero-shot because the virtual forces it experiences match real-world physics.
This approach is particularly vital for contact-rich and deformable object manipulation, where coarse approximations fail. For instance, Du et al. introduce a unified framework in Intersection-free Robot Manipulation with Soft-Rigid Coupled Incremental Potential Contact that guarantees zero interpenetration between soft and rigid bodies. By accurately modeling material deformations and contact surfaces at a mathematical level, the simulator allows policies to transfer directly to physical hardware without requiring extensive domain randomization.
Similarly, high-fidelity modeling is crucial for aerodynamic control. In Weighted Maximum Likelihood for Controller Tuning, the authors show that capturing intricate physical effects like rotor drag and wake aerodynamic interactions within the simulator enables high-speed drone controllers to achieve peak performance immediately upon real-world deployment.
Mitigating the Gap via Action Abstraction
Why should a high-level policy reason at the level of raw motor torques or millisecond-level velocity commands? High-frequency control loops are highly sensitive to the minor latency and friction discrepancies between simulation and reality. A powerful mechanism to bypass this is action abstraction: mapping the policy's output to high-level behavior primitives.
Instead of commanding raw joint velocities every few milliseconds, the policy selects discrete, functional primitives (e.g., "slide until impact" or "grasp object"). Jin demonstrates this in Learning Insertion Primitives with Discrete-Continuous Hybrid Action Space for Robotic Assembly Tasks by abstracting robot actions into primitives that terminate based on physical force feedback rather than strict time steps. This shifts the policy's focus from tracking precise trajectories to navigating physical interaction events.
As Liu validates in Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks, combining specialized functional modules with flexible low-level control makes the overall system significantly less sensitive to the domain gap, maintaining high success rates when deployed on physical hardware.
Exploiting Asymmetric Information
Simulation offers a unique advantage that the real world cannot match: perfect information. During training, the simulator knows the exact 3D coordinates of every object, the friction coefficients, and the full state of the environment. We can leverage this privileged information to accelerate and stabilize learning without requiring it at deployment time.
This is achieved through an Asymmetric Actor-Critic framework. During training, the "Critic" network is given access to the full, privileged state of the simulator to accurately evaluate the agent's actions. Meanwhile, the "Actor" (the control policy) is restricted to processing realistic, deployable sensor streams, such as raw images or tactile data.
Training Phase (Simulation):
[Privileged State] -------> ( Critic ) ---> Evaluates Actions
▲
│
[Sensor Inputs] ----------> ( Actor ) ---> Outputs Control Commands
Deployment Phase (Real World):
[Sensor Inputs] ----------> ( Actor ) ---> Outputs Control Commands
Pinto et al. formalize this in Asymmetric Actor Critic for Image-Based Robot Learning, proving that this asymmetric training setup forces the actor to learn robust visual representations that map directly to optimal actions. Because the critic guides the actor's representation learning behind the scenes, the actor becomes capable of operating purely on raw images when transferred zero-shot to the real world.
Nguyen et al. extend this concept in Leveraging Mutual Information for Asymmetric Learning under Partial Observability, utilizing privileged state information to enhance an agent's information-seeking and memory capabilities under partial observability, ensuring seamless zero-shot deployment even when critical environmental factors are hidden from the robot's physical sensors.