A robot policy trained in simulation usually meets reality in one of two ways. It either acts as a single "robust" controller that has never been told which physics it is in, or it estimates the real physical parameters online (mass, friction, motor gain) and adapts to them. The robust route is domain randomization: training across many randomly perturbed simulators so that the real robot looks like one more random draw. It works, but it tends to produce sluggish, conservative behaviour. The adaptive route recovers performance, but it has a sharp failure mode. Noisy online estimates drift to parameter values where the policy was never trained or never works, and on a robot that falls over without active control, one bad estimate can be enough for a crash.
This note proposes a simple bridge between the two routes. First, use simulation to map out the set of physical parameters on which a parameter-conditioned policy actually succeeds, and fit a convex region inside that set. Then, during deployment, run an online parameter estimator whose every update is projected back onto that region. Projection means replacing an out-of-bounds estimate with the nearest point inside it. The policy is therefore only ever asked to act for physics it has been shown to handle. The claim is that this gives most of the performance of online adaptation while keeping the reliability of a randomized robust policy, including on open-loop unstable hardware where existing guarantees for projected estimators do not apply.
Core Claim
Parameter projection in estimation and control usually uses a feasible set chosen from physical plausibility. Adaptive Economic Model Predictive Control projects least-mean-square (LMS) plant estimates onto a known compact set, and DEPS projects design parameters onto realizable intervals. We propose instead that the set be defined by the policy's demonstrated competence under domain randomization. Let $\xi$ be the physical parameters and $\pi(\cdot \mid \xi)$ a policy trained across a randomization distribution $P_\Xi$ and conditioned on $\xi$. Let $C \subseteq \mathrm{supp}(P_\Xi)$ be a convex set on which Monte Carlo rollouts certify that $\pi(\cdot\mid\xi)$ succeeds with high probability. The deployed estimator is
$$\hat\xi_{t+1} = \Pi_C\big(\hat\xi_t - \eta_t\, g_t\big), \qquad u_t \sim \pi(\cdot \mid s_t, \hat\xi_{t+1}),$$
where $g_t$ is a stochastic gradient of a one-step prediction error on real transitions, $\eta_t$ is a step size, and $\Pi_C$ is Euclidean projection onto $C$. Because $C$ is closed and convex, $\Pi_C$ is non-expansive. So if the true parameter $\xi^*$ lies in $C$, projection never moves the estimate farther from it, and standard projected-LMS convergence arguments carry over. More importantly, at every instant the controller is one that simulation certified on some plant in $C$. If the estimator's step size is small relative to the closed-loop decay rate, a slowly-varying-systems argument then bounds the error of the time-varying loop. That argument, not open-loop stability, is what replaces the AE-MPC assumption.
The case $\xi^* \notin C$ violates the "real world in sim" assumption on which Understanding Domain Randomization for Sim-to-real Transfer builds its bounds. That paper models randomized simulators as latent Markov decision processes and proves sim-to-real gap bounds that require the real dynamics to lie inside the randomization support. When $\xi^* \notin C$, the estimate settles on the boundary of $C$, and whether the boundary policy still controls the true plant becomes a measurable question. We treat it as the main empirical risk of the proposal.
Relation to Prior Work
Four lines of work motivate or constrain the proposal.
Policy Gradient for LQR with Domain Randomization proves convergence of policy gradient for a single linear-quadratic regulator (a linear feedback controller) that jointly stabilizes many sampled linear plants. It uses discount annealing to avoid needing a jointly stabilizing initial gain, and it demonstrates 90% balance success on a Quanser Qube Servo 2 pendulum, where a controller built for the nominal plant (certainty equivalence) fails 9 of 10 times. It motivates the proposal, since randomization clearly beats a nominal model. It also supplies our strongest non-adaptive baseline. It deploys one fixed gain with no projection and no online estimation.
Sim-to-Real Transfer of Robotic Control with Dynamics Randomization reaches 0.89 pushing success with a recurrent policy under dynamics randomization. The recurrent network identifies the parameters implicitly and without constraints, so it is the natural competitor to explicit projected estimation. The latent-MDP theory paper shows such memory is necessary but gives no mechanism that keeps the implicit identification in a competent regime.
DROPO and Auto-Tuned Sim-to-Real Transfer both adapt the simulation distribution toward reality. DROPO fits it offline, from 13 seconds of real data, using the gradient-free optimizer CMA-ES. Auto-Tuned shifts it between training rounds using a learned higher/lower classifier. Both act before or between deployments. Neither projects anything, and neither acts during an episode, which is where our estimator operates.
Provably Safe Sim-to-Real Transfer certifies safety by tracking a tabular mismatch set. Our competence set $C$ is a continuous, parameter-space analogue of "where the simulator can be trusted," but it carries no formal safety guarantee of that kind.
Research Program
The work has three parts.
Build competence sets. For each system we train a parameter-conditioned policy under domain randomization. For the pendulum this is a gain schedule $K(\xi)$ obtained with DR policy gradient at each $\xi$. For richer systems it is a neural policy taking $\xi$ as input. We then evaluate the policy on a dense Monte Carlo sample of parameters. On the points whose success rate exceeds a threshold, we fit a convex inner approximation: either the largest inscribed ellipsoid or a polytope fitted to those points. Ellipsoids keep projection cheap enough to run at control rate, which addresses the computational concern the file raises about high-dimensional projection.
Analyze. We extend the projected-LMS argument of AE-MPC to the setting where the closed loop is stable only through the policy. The certificate is simulation-derived: success on every plant in $C$, plus a bound on how fast $\hat\xi$ can move. The target result is bounded estimation and regulation error when $\xi^* \in C$. For $\xi^* \notin C$, we aim for an explicit dependence on the distance from $\xi^*$ to $C$.
Test. We run the estimator in closed loop on hardware and in cross-engine simulation. The key tests deliberately place the real parameters outside the randomization support.
Baselines
The naive reference is a certainty-equivalence controller built from nominal parameters. On the Qube, Policy Gradient for LQR with Domain Randomization reports that it fails to stay centered in 9 of 10 trials. We rerun it on our own unit rather than borrow that number.
The strongest non-adaptive competitor is the same paper's single DR-PG gain, reported at a 90% balance success rate on the Quanser Qube Servo 2. It is strongest because it is the only method in the file with a hardware result on an open-loop unstable plant under parametric mismatch. We train it on exactly the same $P_\Xi$ used to build $C$, with the same number of simulated rollouts. Beating it means matching its success rate on in-support plants while achieving lower regulation cost. It also means strictly higher success once the real parameters are shifted, for example by added pendulum mass.
The strongest adaptive competitor is an LSTM policy trained under the same randomization, following Sim-to-Real Transfer of Robotic Control with Dynamics Randomization, which reports 0.89 ± 0.06 pushing success on a Fetch arm. For the Qube, the file contains no reported LSTM result. It receives the same simulation budget and the same hyperparameter-search budget as our conditioned policy.
Two ablations isolate the key ingredient. The first uses the identical estimator and policy with no projection, which is adaptive certainty equivalence. The second projects onto the axis-aligned bounding box of the randomization ranges, which amounts to clipping each coordinate. The proposal beats the box ablation only if coupled parameters make the competence set markedly smaller than the box. If it does not, we report that honestly.
Why not just…?
Why not just widen the randomization and deploy one robust policy? Evidence in the file says wide uniform randomization degrades precision and agility. DROPO reports roughly 10 cm pushing error for uniform randomization against roughly 2 cm for a fitted distribution, and Auto-Tuned reports 0% for uniform randomization on both xArm tasks. On the pendulum, however, the single DR-PG gain is a real threat, since it already reaches 90%. Our experiments settle the question by comparing regulation cost and success under widened parameter ranges, where one gain must become more conservative.
Why not just let a recurrent policy identify the physics implicitly? It may work as well in-support, and that would weaken the claim. Its hidden state, however, has no constraint keeping it in a regime the policy handles. The latent-MDP analysis shows that when errors are unrecoverable, the loss can scale with the full horizon. The out-of-support tests below directly measure whether the LSTM or the projected estimator degrades more gracefully.
Why not just clip the estimates to the training ranges? For independent parameters, clipping is exactly projection onto a box, and it may suffice. The box ablation quantifies this. We expect a difference only when parameters trade off: a heavier pendulum with a stronger motor may be fine while a heavier pendulum with a weaker motor is not. We choose shifted-plant scenarios that probe exactly that coupling.
Proposed Experiments
On the Quanser Qube Servo 2 we randomize pendulum mass, length, motor gain, friction and latency. We then test in-support conditions, and out-of-support conditions created by attaching calibrated masses and adding artificial delay. We record balance-and-centering success over repeated trials, quadratic regulation cost, and the fraction of time the unprojected estimate leaves $C$.
For richer dynamics we run cross-engine sim-to-sim tests. We train in Isaac Gym and deploy in a second engine, following the validation protocol of Humanoid-Gym, on a quadruped trotting task. The mismatch here is structural rather than parametric, so it genuinely tests $\xi^* \notin C$.
Three checks are decisive. First, the unprojected estimator should crash measurably more often than the projected one, confirming that estimate excursions cause failures. Second, the competence-set projection should outperform box clipping on at least the coupled-parameter shifts. Third, in out-of-support trials, the projected method should hold success at or above the single DR-PG gain.
If the first check fails, projection is not doing anything. If the second fails, the method reduces to clipping. If the third fails, the "stay inside competence" idea offers no protection exactly when reality leaves the simulator's support. That would be the most informative negative result, and we would report it as such.
References
- Adaptive Economic Model Predictive Control for linear systems with performance guarantees
- Jointly Learning Environments and Control Policies with Projected Stochastic Gradient Ascent
- Understanding Domain Randomization for Sim-to-real Transfer
- Policy Gradient for LQR with Domain Randomization
- Sim-to-Real Transfer of Robotic Control with Dynamics Randomization
- DROPO: Sim-to-real transfer with offline domain randomization
- Auto-Tuned Sim-to-Real Transfer
- Provably Safe Sim-to-Real Transfer
- Isaac Gym: High Performance GPU Based Physics Simulation For Robot Learning
- Humanoid-Gym: Reinforcement Learning for Humanoid Robot with Zero-Shot Sim2Real Transfer