Domain randomization (DR) has become the workhorse of sim-to-real transfer. The intuition is straightforward: if a robot learns to perform a task across a vast range of simulated physical parameters—varying mass, friction, and motor noise—it should be robust enough to handle the specific, unknown parameters of the real world.
However, a growing body of research suggests that this robustness comes at a steep price. By forcing a single policy to succeed across a wide distribution of environments, we often inadvertently train robots to be "cautiously mediocre."
The Lowest Common Denominator Problem
When a reinforcement learning agent is trained via domain randomization, the optimization objective is typically to maximize expected reward across the entire distribution of environments. As Bharadhwaj et al. observe, this often results in a "conservative, average policy" that lacks peak performance.
Imagine training a legged robot to walk on surfaces ranging from ice to sandpaper. To avoid falling on ice, the agent learns to take small, stiff steps. While these steps are safe on sandpaper, they are inefficient and sluggish. Yu et al. argue that these conservative behaviors fail to achieve the agility or precision required for successful real-world operation. The agent prioritizes avoiding failure in edge cases over achieving optimal performance in any specific scenario, a phenomenon Razmjoo et al. describe as prioritizing safety at the cost of specialization.
The Performance-Robustness Tradeoff
This conservative bias creates a fundamental tradeoff. As the range of randomization increases, the policy becomes more robust but less efficient. Candela demonstrates this in autonomous driving, where excessive randomization leads to agents driving at sub-optimal speeds to avoid perceived risks.
In manipulation, the stakes are similar. Lou et al. note that a randomized policy might lack the specific force required to secure a heavy object or, conversely, apply excessive force that displaces a light one. Because the agent doesn't know the object's true mass, it settles on an imprecise middle ground. Dong et al. et al. characterize this brute-force approach as computationally expensive and prone to producing "stiff" robotic behaviors that do not reflect the fluid dynamics of the real world.
Beyond Static Robustness: Online Adaptation
To escape the trap of conservative policies, researchers are shifting from static robustness to dynamic adaptation. The goal is to allow the robot to specialize its behavior to the specific environment it encounters.
One approach is System Identification (SysID), where the robot explicitly or implicitly estimates environmental parameters. Allevato et al. proposed TuneNet to iteratively align simulation parameters with real-world observations, while Murooka et al. developed EXI-Net to condition policies on both explicit physical parameters and implicit latent variables. By identifying the environment, the robot can move from a cautious generalist policy to an optimized specialist policy.
Meta-learning offers another path. Instead of learning one policy, the robot learns how to learn. Song et al. utilize evolutionary strategies to allow legged robots to adapt to failing hardware or slippery floors in minutes. Similarly, Tan et al. introduced Meta Strategy Optimization (MSO) to learn a "strategy space" that enables rapid adaptation in fewer than 15 trials.
Intuition: From Safety to Agility
The shift away from broad domain randomization is essentially a shift in how we handle uncertainty.
- Domain Randomization treats uncertainty as a fixed noise distribution to be averaged out, leading to cautious, "safe" behavior.
- Adaptation treats uncertainty as a variable to be resolved through interaction.
As Kumar et al. show with Rapid Motor Adaptation (RMA), a robot that can estimate environmental conditions from its own movement history can navigate mud, sand, and stairs with an agility that static, conservative policies cannot match. By moving beyond the "average" solution, these methods allow robots to regain the peak performance lost to the reality gap.
Key Foundational Work
Learning Fast Adaptation with Meta Strategy Optimization (2019) — contributed 2 concept elements to this direction.
EXI-Net: EXplicitly/Implicitly Conditioned Network for Multiple Environment Sim-to-Real Transfer (2020) — contributed 1 concept elements to this direction.
Sim-to-Real Transfer for Biped Locomotion (2019) — contributed 1 concept elements to this direction.