Robots are often trained in a simulator whose physical settings, such as friction, mass, damping or rope length, are wrong. One effective fix runs a loop. Train a policy in simulation, run it briefly on the real robot, use a learned classifier to decide whether each simulator parameter should go up or down, and repeat. Auto-Tuned Sim-to-Real Transfer showed this loop reaching 90% real success on a cabinet-sliding task, where domain randomization (training over wide random parameter ranges) scored 0%. Nobody knows how many expensive real-world rounds the loop needs, or when it will fail. Its authors tested mostly one-dimensional or decoupled parameters and report that it diverges from poor initial guesses.
This note proposes to read the loop as an optimization method that receives only noisy "higher or lower" answers, and to analyze its convergence rate the way optimization theory analyzes iterative solvers. The central claim is that the number of real rounds is governed by two measurable quantities. The first is how often the classifier's direction answer is correct, which we call its margin. The second is how strongly the parameters are coupled in their effect on observed motion. A per-coordinate up/down search is provably fragile under coupling. A small change, asking the classifier about random directions in parameter space rather than one axis at a time, restores a geometric (linear-rate) decrease in parameter error. Its round count can be predicted before running on hardware.
Core Claim
Let $\theta$ be the simulator parameter vector in $d$ dimensions and $\theta^*$ the unknown real values. Near $\theta^*$, the real-versus-simulated difference in observed motion behaves like $A(\theta - \theta^*)$ for some sensitivity matrix $A$. An axis-wise classifier of the kind used by Auto-Tuned effectively reports the sign of each entry of $A(\theta-\theta^*)$, not of $\theta-\theta^*$ itself. When $A$ is far from diagonal, those signs can point the wrong way, and step-halving search can cycle. Mass and damping, for example, both slow a swinging link.
We propose instead to train the classifier to answer "is $\theta^*$ on the positive side of direction $u$ from $\theta$?" for directions $u$ drawn at random. This turns the classifier into a comparison oracle whose expected response is aligned with $\theta^*-\theta$ under a mild condition on $A$. Suppose the classifier is correct with probability $1/2+\gamma$, where $\gamma>0$ is its margin. Combining several direction queries per round with a geometrically shrinking step should give
$$\|\theta_K - \theta^*\| \le \varepsilon \quad \text{once} \quad K \gtrsim \frac{c\,\kappa(A)\, d}{\gamma^2}\,\log\frac{R_0}{\varepsilon},$$
where $K$ is the number of real deployment rounds, $R_0$ the initial parameter error, $\kappa(A)$ the conditioning of the sensitivity matrix, and $c$ a constant. The form follows from standard reasoning, not a finished proof. The $1/\gamma^2$ factor is the Hoeffding cost of majority-voting a biased coin. The $d\,\kappa(A)$ factor is the number of random directions needed to recover a direction estimate correlated with the true error. The logarithm is the usual signature of a constant-factor contraction per round.
A testable corollary follows. When the classifier's accuracy approaches 50%, $\gamma \to 0$ and no step schedule makes progress. This mirrors the empirical finding in Adaptive Training Distributions with Scalable Online Bilevel Optimization, where data selection stopped helping on YFCC15M once their gradient-agreement diagnostic hovered near 50%. Our margin $\gamma$ plays the same diagnostic role for calibration.
Relation to Prior Work
Auto-Tuned Sim-to-Real Transfer supplies the method we analyze and modify. It has no convergence guarantee, no round count, and no treatment of coupled parameters, and we add all three.
The rate argument borrows its shape from error-bound analyses. Convergence rate of a gradient projection method for solving variational inequalities derives explicit linear factors from a constant bounding distance-to-solution by a computable residual. Such constants are usually impossible to evaluate in practice. In our setting the analogous quantities, $\gamma$ and $\kappa(A)$, can be estimated cheaply from simulated rollouts before touching hardware, because in simulation the ground truth is known.
An Adaptively Inexact First-Order Method for Bilevel Optimization with Application to Hyperparameter Learning constrains the alternative of treating calibration as gradient-based bilevel optimization. Its guarantees are asymptotic only and require a strongly convex inner problem. Policy learning in simulation violates that, which is one reason we work with sign/comparison feedback rather than hypergradients.
Provably Safe Sim-to-Real Transfer gives a real-sample complexity scaling with the size of the mismatch region. That result is tabular, assumes mismatches are separated by a fixed margin, and does not count optimization rounds. Our object is different: continuous parameters, a learned visual oracle, and a bound on calibration rounds rather than exploration samples.
Research Program
The work has three parts.
The first is theory. We will prove the contraction bound above for the random-direction comparison scheme under a local linearity assumption on $A$ and an independence-of-errors assumption on the classifier. We will also construct explicit two-parameter counterexamples in which axis-wise sign search cycles whenever $A$ fails a diagonal-dominance condition. Both results will be checked against controlled simulations in which $A$ is known.
The second is the algorithm. We will train the existing Search Param Model architecture on direction-labelled synthetic rollouts. Each round queries a fixed budget of random directions on the same short real video clip, so no extra real data is collected per query. The step size shrinks by a factor set from the estimated $\gamma$, rather than by hand.
The third is a predictive check. Before any real deployment, we estimate $\gamma$ and $\kappa(A)$ in simulation, predict the number of rounds, then compare the prediction to what actually happens.
Baselines
The naive reference is domain randomization. We randomize every parameter over the same prior box used to initialize calibration, train with the same total simulation-step budget, and do no real-world rounds. Auto-Tuned Sim-to-Real Transfer reports it at 0% success on both Cabinet Slide and Rope Peg-in-Hole, so it anchors the low end of success. For parameter-error metrics it has no calibrated estimate at all.
The strongest existing method is the original Auto-Tuned axis-wise search. It reports 90% real success on Cabinet Slide and 40% on Rope Peg-in-Hole on the xArm 7. It is the right competitor because it uses exactly the same feedback type: a short real clip and a learned classifier. We will run it with the same classifier backbone, the same real clips per round, the same policy learner, and an equal tuning budget for its step schedule. We will not inherit its published schedule unchanged.
A third baseline is a black-box optimizer such as Bayesian optimization or CMA-ES. Its objective is a learned distance between real and simulated clips, and it gets the same number of real rounds. No reported result was found on Lacuna for this baseline on these tasks.
The key ablation isolates direction queries. It keeps our step schedule but reverts to axis-wise signs, and separately keeps random directions but uses the original schedule.
Beating the strongest method means two things. On the four-to-eight-dimensional coupled settings, we must reach a given parameter error in significantly fewer real rounds, across seeds. We must also be no worse on the one-dimensional tasks where Auto-Tuned already works.
Why not just…?
Why not just use domain randomization? It needs no real rounds, but the cited evidence is a 0% success rate on contact-rich tasks where calibration reached 90% (Auto-Tuned Sim-to-Real Transfer). Randomizing over coupled parameters also widens the training distribution multiplicatively. We include it in every experiment, so the gap will be measured, not assumed.
Why not just run Bayesian optimization or CMA-ES over simulator parameters? This is a real threat in low dimension. With two parameters a good surrogate model may match any sign-based method. Our claim is restricted to the regime where $d \ge 4$ and parameters are coupled, where surrogate models need many evaluations and each evaluation is a real deployment. The black-box baseline is run at matched real-round budgets across $d \in \{1,2,4,8\}$. If it matches us at $d=8$, the algorithmic contribution shrinks to the analysis alone, and we will report that.
Why not just keep the original axis-wise search and run more rounds? Under strong coupling, more rounds do not help if the search cycles. The two-parameter counterexamples and the ablation test this directly. If axis-wise search converges as fast in practice on Control Suite coupling, our fragility claim is wrong.
Why not just fine-tune the policy on real data and skip calibration? Fine-tuning needs far more real interaction than a few short clips, and it carries safety risk. That risk motivated Provably Safe Sim-to-Real Transfer, whose unconstrained transfer baseline was unsafe in 68.5% of iterations on its gridworld. We will report real interaction steps for a fine-tuning baseline at matched success on the sim-to-sim tasks.
Proposed Experiments
The main testbed is sim-to-sim on the DeepMind Control Suite tasks used by Auto-Tuned: Cheetah Run, Walker Walk, Finger Spin and Ball in Cup. Here a held-out "real" simulator has hidden parameters, so parameter error is measurable exactly. No reported numbers for these tasks were found on Lacuna. We will vary the number of calibrated parameters from 1 to 8 and deliberately include coupled pairs, such as link mass with joint damping and actuator gain with mass. We report real rounds to reach a given $\varepsilon$, final parameter error, and transferred episode return.
The decisive check has two parts. On log axes, does measured round count track the predicted $d\,\kappa(A)/\gamma^2 \cdot \log(1/\varepsilon)$? Does the predicted failure appear when we artificially degrade classifier accuracy toward 50%?
If hardware is available, we then repeat Cabinet Slide on the xArm 7 with two added coupled parameters. The target is to match the reported 90% with fewer real rounds. The idea fails if the predicted and observed round counts are uncorrelated, or if axis-wise search shows no cycling or slowdown under measured coupling.
References
- Auto-Tuned Sim-to-Real Transfer
- Adaptive Training Distributions with Scalable Online Bilevel Optimization
- Convergence rate of a gradient projection method for solving variational inequalities
- An adaptively inexact first-order method for bilevel optimization with application to hyperparameter learning
- Provably Safe Sim-to-Real Transfer