Introduction
Reinforcement Learning (RL) has demonstrated remarkable success in simulated environments, yet the "reality gap"—the discrepancy between simulation and the physical world—remains a significant barrier to deploying these policies on real robots. Domain Randomization (DR) is a common technique used to address this, where a policy is trained across a distribution of simulated physics parameters (e.g., friction, mass) to ensure robustness. However, manually tuning these distributions is often a tedious "black art." If ranges are too narrow, the policy fails on the real robot; if they are too wide, the task becomes unlearnable.
The DROPO Framework
The authors propose DROPO (Domain Randomization Off-Policy Optimization), an automated framework that identifies optimal physics distributions using a minimal amount of offline real-world data. Unlike online methods that require continuous robot interaction—risking hardware damage and slowing down training—DROPO leverages pre-collected trajectories to tune the simulator, creating robust policies that transfer to physical robots with zero-shot success.
The process begins with a small dataset of real-world trajectories, which can be collected via human kinesthetic teaching or simple controllers. These trajectories serve as the "ground truth" for optimizing the simulator's parameters.
Technical Methodology: Likelihood-Based Optimization
At the core of DROPO is a shift from traditional distance-based (L2) error minimization to a likelihood-based objective. Traditional System Identification (SysID) often seeks a single "best" value for a parameter, which can lead to distribution collapse. In contrast, DROPO treats the simulator as a stochastic forward model and calculates the probability that real-world trajectories could have been generated by specific simulated environments.
Key features of the methodology include:
- Variance Modeling: DROPO optimizes both the mean and the variance of dynamics parameters. This allows the system to use variance as a buffer for unmodeled physical phenomena. If the simulator lacks a specific effect (like air resistance), DROPO naturally widens the distribution of other parameters to "soak up" the unexplained reality.
- Black-Box Optimization: Utilizing CMA-ES, a gradient-free optimizer, DROPO remains compatible with standard simulators like MuJoCo without requiring complex differentiable physics engines.
- The $\epsilon$ Hyperparameter: A noise term ($\epsilon$) acts as a regularizer. The authors found they could tune this parameter using an "elbow method" on Mean Squared Error plots, providing a principled way to balance system identification and randomization.
Experimental Evaluation
The framework was evaluated across three distinct scenarios: a sim-to-sim verification using the Hopper environment, a HockeyPuck task using a Kuka LWR4+ robot arm, and a Pushing task using a Franka Panda robot. In the pushing task, the robot had to handle a box with an asymmetric, unmodeled center of mass.
Key Results
In the Franka Panda pushing task, DROPO demonstrated exceptional precision, achieving a target distance of approximately 2cm. This significantly outperformed baselines like DROID (~7cm) and standard Uniform Domain Randomization (~10cm). Notably, the system proved highly data-efficient, requiring only 13 seconds of real-world interaction data to find effective distributions. Furthermore, in tests where physical parameters were intentionally hidden from the optimizer, DROPO was the only method that successfully adapted its remaining parameters to maintain high performance.
Conclusion and Future Work
DROPO provides a robust, data-efficient solution for bridging the reality gap. By explicitly modeling parameter variance through a likelihood-based lens, it creates policies that are resilient to the uncertainties of the physical world. While the current implementation handles up to 18 parameters, future work will explore scaling to higher-dimensional spaces and extending the framework to vision-based policies, which would be a significant step forward for digital twin applications.
Summary updated