Abstract
We study the sample efficiency of domain randomization and robust control for the benchmark problem of learning the linear quadratic regulator (LQR). Domain randomization, which synthesizes controllers by minimizing average performance over a distribution of model parameters, has achieved empirical success in robotics, but its theoretical properties remain poorly understood. We establish that with an appropriately chosen sampling distribution, domain randomization achieves the optimal asymptotic rate of decay in the excess cost, matching certainty equivalence. We further demonstrate that robust control, while potentially overly conservative, exhibits superior performance in the low-data regime due to its ability to stabilize uncertain systems with coarse parameter estimates. We propose a gradient-based algorithm for domain randomization that performs well in numerical experiments, which enables us to validate the trends predicted by our analysis. These results provide insights into the use of domain randomization in learning-enabled control, and highlight several open questions about its application to broader classes of systems.
Introduction
In modern robotics, Domain Randomization (DR) has emerged as a "cheat code" for bridging the sim-to-real gap. By training controllers on a wide variety of randomized simulated environments—varying parameters like friction, mass, or geometry—practitioners have successfully deployed complex behaviors on physical hardware. Despite its empirical dominance, the mathematical foundations of DR have remained elusive. This paper provides a theoretical analysis of DR’s sample efficiency, comparing it against two traditional paradigms: Certainty Equivalence (CE) and Robust Control (RC).
Control Paradigms under Uncertainty
When learning a world model from data, three primary strategies exist to handle the inherent uncertainty of the estimated parameters:
- Certainty Equivalence (CE): This approach treats the estimated model as the ground truth. While computationally efficient and asymptotically optimal, it is prone to instability in the "low-data regime" where model estimates are noisy.
- Robust Control (RC): This strategy optimizes for the worst-case scenario within a set of possible models. While it guarantees stability, it is often overly conservative and computationally demanding, typically requiring the solution of complex Linear Matrix Inequalities (LMIs).
- Domain Randomization (DR): DR minimizes the expected (average) cost over a distribution of models. This paper formalizes DR as a middle ground that captures the benefits of both CE and RC.
Theoretical Framework: The LQR Benchmark
The authors analyze these methods using the Linear Quadratic Regulator (LQR), a fundamental problem in control theory. The goal is to determine how the "excess cost"—the performance gap between a learned controller and the optimal controller—decays as the number of samples ($N$) increases.
Asymptotic Optimality
A key contribution of this work is the proof that DR is asymptotically optimal. The authors show that if the randomization distribution is chosen strategically—specifically, a uniform distribution over a confidence ellipsoid derived from the Fisher Information of the data—DR achieves the same $1/N$ rate of decay in excess cost as Certainty Equivalence. This confirms that DR does not sacrifice long-term performance for its practical flexibility.
The Burn-in Advantage
While CE and DR share the same asymptotic rate, their behavior in the low-data regime differs significantly. CE often suffers from a long "burn-in" period, where it requires a substantial amount of data before it can produce a stabilizing controller. In contrast, DR and RC account for uncertainty from the outset, allowing them to stabilize systems with far fewer data points. This explains why DR is often more reliable in real-world robotics applications where data collection is expensive or dangerous.
Algorithmic Innovation: Gradient-Based DR
To make DR scalable, the authors introduce a Domain Randomized Policy-Gradient algorithm. Unlike Robust Control, which struggles to scale to high-dimensional or nonlinear systems due to its reliance on LMIs, this DR algorithm is "gradient-friendly." It samples model scenarios and performs gradient descent on the controller parameters, specifically focusing on scenarios where the current controller remains stable. This approach bridges the gap between theoretical control theory and the practical requirements of deep reinforcement learning.
Experimental Validation
The theoretical findings were validated through two primary benchmarks:
- 3D Linear System: Used to confirm the $1/N$ decay rates and the performance gap between CE, RC, and DR. The results showed that while RC remains conservative even with infinite data, DR eventually converges to the optimal performance of CE.
- Nonlinear Pendulum: To demonstrate the broader applicability of their insights, the authors applied DR to a nonlinear pendulum using Receding Horizon Control. The experiments confirmed that DR provides superior stability and performance compared to CE when the model is not perfectly known.
Conclusion and Future Work
This research provides a rigorous justification for the use of Domain Randomization in control. It proves that DR is not merely a heuristic but a sample-efficient method that matches the asymptotic performance of Certainty Equivalence while offering the stability benefits of Robust Control.
However, several challenges remain. The current theoretical bounds for DR's burn-in time are still conservative and do not yet fully reflect the superior performance seen in experiments. Furthermore, extending these guarantees to fully nonlinear systems and addressing "model misspecification"—where the true system lies outside the randomization distribution—are critical avenues for future research.
Summary updated