Is Sim-to-Real Calibration Curved Where It Matters? Measuring Quadratic Growth on Real Targets

Before a simulator can be trusted to stand in for a real robot, its physical constants have to be calibrated: friction, solver timestep, actuator limits, and dozens more, chosen so that simulated trajectories match recorded real ones. Today this search is treated as a blind black box. It is slow (one study needed 279 to 662 hours per physics engine), and it gets worse as more parameters are added. Optimization theory has a precise vocabulary for why some searches converge fast even without a unique answer. One example is quadratic growth: the error rises at least quadratically as you move away from the set of best parameters, even if that set is a whole flat valley. That vocabulary has never been applied to calibration. It also carries a methodological flaw of its own. The papers that prove fast convergence under these relaxed conditions check them almost only on clean synthetic data.

We propose to measure the curvature of calibration landscapes directly. The key comparison is between targets the simulator itself generated (the standard "sim-to-sim" test) and targets recorded from real hardware. Our claim has two parts. First, calibration error is sharply curved along a small number of parameter directions and flat along the rest, so a curvature-aware method restricted to the curved directions should converge at a linear (geometric) rate where black-box search stalls. Second, synthetic targets systematically overstate that curvature. On real targets the simulator cannot reproduce the data exactly, so the fast rate gives way to a floor. When an optimizer's observed contraction rate breaks down, that breakdown is a measurable signature of the reality gap.

Core Claim

Let $\theta$ be the simulator parameters, and let $F(\theta)$ be the mean squared distance between simulated and recorded end-effector trajectories. Let $F^*$ be the best achievable value and $\Theta^*$ the set of parameters that achieve it. The condition we test is restricted quadratic growth:

$$F(\theta) - F^* \ge \frac{\mu}{2}\,\mathrm{dist}(P_S\theta, P_S\Theta^*)^2,$$

Here $P_S$ projects onto a low-dimensional subspace $S$ of parameter directions, and $\mu>0$ is the curvature constant on that subspace. This mixes two ideas from the optimization literature. One is quadratic growth toward a non-unique optimal set, the condition under which randomized coordinate descent converges linearly and under which adaptive Frank–Wolfe methods converge locally linearly. The other is restricted curvature along a structured subspace, as in linear convergence of stochastic hard-thresholding under restricted strong convexity. Suppose the condition holds and $F$ has $L$-Lipschitz gradients along $S$. A gradient-type method run in $S$ then satisfies $F(\theta_k)-F^* \le (1-\mu/L)^k\,(F(\theta_0)-F^*)$. This is the standard contraction argument, which uses the growth condition in place of strong convexity.

The empirical prediction is specific. In Traversing the Reality Gap via Simulator Tuning, three parameters (timestep, lateral friction, maximum joint velocity) dominated fidelity, and expanding from 31 to 57 parameters added noisy local minima. That pattern is exactly what a few curved directions plus many near-flat, poorly identified ones would produce. We further predict that $\mu$ estimated against synthetic targets will exceed $\mu$ estimated against real targets, and that the gap will widen as the target becomes less reproducible by the simulator. The first part of the claim, curvature, motivates a faster calibrator. The second part, synthetic overstatement, is the contribution about evaluation, and it stands even if the faster calibrator does not.

Research Program

The work has three pieces.

The first is a curvature probe. At a handful of parameter settings we estimate the Jacobian of the trajectory residual by finite differences, averaging several rollouts per perturbation to tame contact noise. From it we form the Gauss–Newton curvature matrix, whose eigenvalues show which directions the data actually constrain. The subspace $S$ is spanned by eigenvectors whose eigenvalues stay above a threshold across probe points. We then estimate $\mu$ from the ratio $(F(\theta)-\hat F^*)/\mathrm{dist}^2$ along sampled segments toward the best point found. On landscapes small enough to evaluate exhaustively, we compute this ratio exactly.

The second is a calibrator: Levenberg–Marquardt steps (Gauss–Newton with damping) restricted to $S$. The flat directions are frozen at default values, since by construction the data cannot pin them down. The calibrator tracks its empirical contraction ratio $(F(\theta_{k+1})-\hat F^*)/(F(\theta_k)-\hat F^*)$ at every step.

The third, and the methodological core, is a protocol that compares a ladder of targets. The bottom rung is the same engine with known parameters, which makes the calibration problem well-specified. The middle rung is a different engine, which introduces controlled misspecification. Collins et al. already show that Bullet, ODE and Newton disagree systematically, so this rung needs no new physics. The top rung is real recorded trajectories. On each rung we report $\mu$, the dimension of $S$, the observed contraction rate, and the residual floor $F^*$. The diagnostic we test is whether a sustained rise of the contraction ratio toward 1, at a floor well above sensor noise, reliably flags misspecification.

Baselines

The naive reference point is the engine's default parameters with no calibration. Collins et al. report that tuning reduces trajectory error by 14% to 91% relative to this default on the Kinova benchmark, so any calibrator must at least land inside that band.

The strongest existing calibrator for trajectory matching is Differential Evolution with the best1bin strategy over shared and individual physics parameters, as in Traversing the Reality Gap via Simulator Tuning. It is the method actually used on this exact problem, and it reports 279 hours of optimization for Newton and 662 for PyBullet. We rerun it on identical tasks, engines and target trajectories. Budgets are counted in simulator rollouts, not wall-clock hours, so that finite-difference probing is charged in full. Beating it means reaching equal or lower held-out trajectory error with at most a third of the rollouts, or reaching lower error at equal rollouts.

For closed-loop transfer, the strongest competitor is the Search Param Model of Auto-Tuned Sim-to-Real Transfer. It learns from synthetic video whether each parameter is above or below its real value, and reports 90% success on Cabinet Slide and 40% on Rope Peg-in-Hole, against 0% for domain randomization, using under 20% of domain randomization's simulation steps. We run it with its published setup on the same task, and give our calibrator the same simulation-step budget. Beating it means matching its policy success on Cabinet Slide at fewer steps, or improving on its 40% on Rope Peg-in-Hole. We also run CMA-ES and Bayesian optimization at matched rollout budgets as generic black-box competitors; no reported result was found on Lacuna for either on these benchmarks.

Two ablations isolate the key ingredient. The first runs the same Levenberg–Marquardt calibrator over the full parameter space, which tests whether restricting to $S$ matters. The second uses a subspace chosen by one-shot sensitivity at the default parameters rather than by curvature that persists across probe points, which tests whether the persistence requirement matters.

Why not just…?

Why not just use domain randomization and skip calibration? The file's direct evidence is that it can fail outright: 0% success on both xArm tasks in Auto-Tuned Sim-to-Real Transfer, where calibrated parameters reached 90% and 40%. We include domain randomization in the xArm experiment as a reference, but we do not expect it to compete.

Why not just run a better black-box optimizer such as CMA-ES or Bayesian optimization? This is the real threat to the speed claim. If calibration landscapes really are curved along a few directions, CMA-ES adapts to that shape on its own, and Bayesian optimization may handle a few dominant parameters well. The matched-rollout comparison settles it. If CMA-ES matches our calibrator, we drop the speed claim and keep the evaluation claim. The curvature measurements and the synthetic-versus-real gap in $\mu$ do not depend on which optimizer wins.

Why not just use a differentiable simulator and plain gradient descent? Contact makes simulator gradients unreliable. More importantly, the paper's own evidence of noisy local minima as parameters are added suggests the problem lies in the landscape, not in how gradients are obtained. We test this by running gradient descent on smoothed finite-difference gradients over the full space, which is essentially the first ablation. If it matches the restricted calibrator, the subspace restriction is unnecessary.

Why not just do a standard parameter-sensitivity or identifiability analysis? Sensitivity analysis and our probe overlap heavily, and we say so; the second ablation is exactly that. What sensitivity analysis does not give is a convergence certificate, a stopping rule tied to $\mu$, or the comparison across the ladder of targets. The latter is where the new claim lives.

Proposed Experiments

The first study is cheap and fully exhaustive. Is this Model for Real? evaluated Bayesian Knowledge Tracing fit (log-likelihood over Prior, Learn, Guess and Slip parameters) on a grid of 390,625 parameter combinations for 42 question groups from ASSISTments, on both real and simulated student data. It reports Learn values clustering under 0.12 and high Guess and Slip rates (above 0.5) appearing equally in real and simulated data. On the full grid we can compute the optimal set, $\mu$ and the flat directions exactly for each group, real and simulated. The decisive check: if simulated $\mu$ is not systematically larger than real $\mu$ across groups, the overstatement hypothesis fails at its simplest test.

The second study is cross-engine. We use the Kinova-style task family of free-space motion, pushing, and rolling cubes, cylinders and cones. Targets come from one engine with known parameters and then from a different engine. We measure $\mu$, the dimension of $S$, and rollouts to a fixed error against Differential Evolution, CMA-ES and Bayesian optimization. If $S$ does not recover something close to the timestep, friction and velocity directions that Collins et al. found dominant, the curvature story is wrong.

The third study uses real targets: newly recorded motion-capture trajectories at 20 Hz on the same task family, reusing the original Kinova data if it is available. We add the xArm Cabinet Slide and Rope Peg-in-Hole tasks for closed-loop success against the Search Param Model. The decisive checks are whether the contraction-rate breakdown predicts held-out trajectory error better than the raw training residual does, and whether calibration on the restricted subspace generalizes across tasks better than full-space tuning. The second check addresses the known problem that tuning friction on one trajectory distorts other interactions. A clean negative outcome on all three studies, with no curvature gap between synthetic and real targets and no rollout savings, would itself be informative. It would mean the reality gap here behaves as noise rather than as lost curvature.