Treating the Simulator as a Biased Gradient: Setting Sim-and-Real Mixing Weights from Measured Gradient Bias

When a robot policy or a perception model is trained on a large simulated dataset plus a small real one, the most consequential choice is how much weight each source gets, and when. Today that weight is set by trial and error. Each sim-to-real method, whether fine-tuning, fixed-ratio co-training, domain randomization or adversarial feature alignment, arrives with its own story and its own tuning knobs. We propose a single lens for this choice. A gradient computed on simulated data is a cheap, low-noise estimate of the gradient of the real-world loss, but it is biased. A gradient computed on the few real examples is unbiased but noisy. The optimal mixing weight is the one that minimizes the error of the combined gradient. That weight can be estimated online from quantities measurable during training, separately for each part of the network.

The practical bet is that this estimate, applied per block of layers, does what current methods achieve with adversarial machinery. Shared perception layers would take mostly simulated gradient wherever simulation and reality agree. Layers that encode physics, such as the policy head that must output real contact forces, would shift toward real gradient wherever they disagree. This is the "structured aligned" geometry that a recent analysis of co-training identified as the one that transfers. The analytical claim is equally concrete: measured gradient bias should predict real-world transfer better than feature-distribution distance does.

Core Claim

Let $F_r(\theta)$ be the real-world loss and $F_s(\theta)$ the simulated loss of parameters $\theta$. Let $b(\theta)=\nabla F_s(\theta)-\nabla F_r(\theta)$ be the simulator's gradient bias. Let $\sigma_s^2, \sigma_r^2$ be per-example gradient variances, and $m_s, m_r$ the per-step batch sizes. The mixed step $g_w = w\,\hat g_s + (1-w)\,\hat g_r$ then has mean-squared error against $\nabla F_r$ of $w^2\|b\|^2 + w^2\sigma_s^2/m_s + (1-w)^2\sigma_r^2/m_r$. This error is minimized at

$$ w^\star(\theta) = \frac{\sigma_r^2/m_r}{\|b(\theta)\|^2 + \sigma_s^2/m_s + \sigma_r^2/m_r}. $$

Standard biased-SGD analysis bounds the average squared real gradient norm after $T$ steps of size $\eta$ on an $L$-smooth $F_r$. Up to constants, the bound is $O\big((F_r(\theta_0)-F_r^\star)/(\eta T)\big)$ plus the variance and squared-bias terms above. Choosing $w$ to minimize that error therefore minimizes the non-vanishing floor of the bound.

Everything on the right-hand side is estimable. $\|b\|^2$ is estimated without bias by $\|\hat g_s-\hat g_r\|^2-\sigma_s^2/m_s-\sigma_r^2/m_r$, with variances taken from within-batch per-example gradients. The real gradient for this estimate comes from a held-out real split, so the bias reflects the real population and not the 50 memorized demonstrations. Applying the formula per parameter block $k$ gives weights $w_k^\star$.

This one formula subsumes the existing heuristics as weight schedules:

  • sim-pretrain-then-fine-tune is $w$ jumping from 1 to 0;
  • fixed-ratio co-training is constant $w$;
  • domain randomization and alignment regularizers act by changing $b$ itself.

Where This Sits in Prior Work

The optimization side supplies the method. Optimization with First-Order Surrogate Functions unifies proximal, coordinate and Frank-Wolfe methods as minimizing surrogates whose error relative to the true objective has a Lipschitz gradient. Our framing reads the simulator loss as exactly such a surrogate, except that its error is not controlled by design and must be measured. Plugin analyses such as A Unified Convergence Theorem for Stochastic Optimization Methods and Taming the Wild: A Unified Analysis of Hogwild-Style Algorithms show how one master argument absorbs many noise sources. Those analyses assume the noise is unbiased or bounded hardware error. We add a distribution-shift bias term that the file notes these frameworks never model.

A Unified and General Framework for Continual Learning is the closest stylistic precedent. It shows regularization, Bayesian and replay methods are parameterizations of one Bregman-divergence objective. It unifies loss functions. We unify step-weighting decisions, and we derive an operational rule rather than a taxonomy.

The sim-to-real side supplies the motivation. A Mechanistic Analysis of Sim-and-Real Co-Training in Generative Robot Policies derives the optimum of co-trained diffusion policies. It finds three latent regimes: disjoint, overlapping (blind alignment that erases physics), and structured aligned. Its CFG-ADDA method combines adversarial alignment with domain-conditioned guidance. That paper also leaves the mixing weight to manual tuning and gives no convergence rates; we target both gaps directly.

Two papers constrain the design.

Both push features together. The co-training analysis warns that this can destroy physical cues, so our method deliberately never penalizes feature distance and lets each block decide from its own gradients.

Distributionally Robust Learning for Unsupervised Domain Adaptation reweights examples by density ratio. That ratio degenerates in high dimensions. We reweight parameter blocks by gradient error, which needs no density estimate, though it does need some labeled real data.

Gradient-similarity heuristics from multi-task learning, which gate an auxiliary task's gradient by cosine agreement, are the nearest generic relatives. They are not represented in this file. We treat cosine gating as an ablation rather than claim distance from it.

Research Program

The work has three parts.

Estimator and optimizer. We build an estimator of $\|b_k\|^2$, $\sigma_{s,k}^2$ and $\sigma_{r,k}^2$ per block. It uses exponential moving averages to tame noise from a 50-example real set. We prove the biased-oracle rate for per-block weights under smoothness. We also state plainly what the rate does not cover: generalization from a tiny real set, which the held-out bias estimate only partly addresses.

Analysis instrument. The same measurement doubles as a diagnostic. For every existing method we log per-block gradient bias along training and test whether it explains transfer. The co-training paper reports a correlation of 0.6 to 0.8 between Wasserstein feature alignment and performance. We predict that bias in the policy-head blocks correlates more strongly. We also predict that "overlapping" runs show low feature distance but high head-block bias.

Generality. We apply one unchanged procedure to two very different problems: diffusion-policy manipulation and LiDAR semantic segmentation.

Baselines

No mixing. Real-only training and source-only training are the naive reference points. On SynLiDAR to SemanticKITTI without target labels, source-only is reported at 15.2% mean intersection-over-union (mIoU), and a real-data oracle at 55.5%, in Adversarially Masking Synthetic to Mimic Real.

Fixed-ratio co-training with $w$ grid-searched on a real validation split. This is the practitioner's default and the most direct competitor to an automated weight. It gets the same data and backbone, a five-to-seven point grid, and its evaluation cost is counted in real rollouts.

Sim-pretrain then real fine-tune. This runs with the same total steps.

CFG-ADDA from the co-training paper. This is the strongest existing method. It reports a 74% average real-world success rate on NutAssembly, MugHang and MugCleanup, roughly 20% above standard approaches, using 3,000 MimicGen simulated and 50 real trajectories. We rerun it with the authors' published hyperparameter recipe and the same tuning budget we give ourselves. Beating it means matching its success rate without an adversary or guidance, or exceeding it when combined with it.

Synthetic-data integration on the segmentation side. The strongest semi-supervised reference is SynLiDAR. It reports improving SemanticKITTI mIoU from 60.3% to 64.7% by adding translated synthetic scans, and matching baseline accuracy on SemanticPOSS with 40% less labeled real data. We reproduce it on the same backbone and label fractions.

Ablations isolating the key ingredient.

  • Global versus per-block weights.
  • The MSE-optimal weight versus cosine-agreement gating.
  • Bias estimated on held-out versus training real data.

Why not just…?

Why not just grid-search the mixing weight on real validation data? This is the real threat. If the best constant $w$ matches our schedule, the contribution reduces to saving tuning cost. That saving still matters when each validation point means dozens of real robot rollouts, but it is a smaller claim. Two things settle the question. First, we compare against the grid-searched constant in a controllable proxy where evaluation is free, so the oracle can be found exactly. Second, we test whether per-block weights beat every constant. The theory predicts they will when bias is concentrated in a few blocks. If they do not, we will report that per-block structure adds nothing.

Why not just pretrain on simulation and fine-tune on real data? That is the $w$: 1→0 schedule inside our family. Its weakness is that the jump is global and abrupt: all blocks lose the low-noise simulated gradient at once. The per-block run shows directly whether perception layers keep benefiting from simulation after the head has switched to real data.

Why not just use adversarial alignment such as CFG-ADDA? CFG-ADDA works, and its 74% is the bar. It does, however, need a discriminator and guidance machinery, and the alignment literature documents discriminators latching onto sensor-dropout shortcuts (Adversarially Masking Synthetic to Mimic Real). We test whether bias-weighting alone reaches the structured-aligned regime, measured by the paper's linear-probe discernibility and Wasserstein metrics. We also test whether adding it on top of CFG-ADDA helps. If CFG-ADDA alone wins, the diagnostic contribution still stands, but the method claim fails.

Why not just widen domain randomization? Randomization changes the simulated distribution. It does not decide how to mix it with real data, so it is complementary. We run randomized and unrandomized simulators under each weighting rule and expect randomization to show up as reduced measured bias. Randomization's interpolation-only guarantee (Robust Domain Randomization for Reinforcement Learning) predicts that bias stays high when the real parameters fall outside the randomized range.

Proposed Experiments

Controlled manipulation proxy. In robosuite, with NutAssembly, MugHang and MugCleanup, a "real" environment is the simulator with shifted friction, mass and camera pose at graded magnitudes. We train diffusion policies on 3,000 MimicGen trajectories plus 50 "real" ones. Because ground truth is cheap here, we can compute the true optimal constant $w$ and true bias. The decisive checks are:

  • whether the estimated $w^\star$ tracks the oracle constant as the shift grows;
  • whether per-block weighting exceeds the best constant;
  • whether head-block bias predicts success rate better than the reported 0.6 to 0.8 feature-alignment correlation.

Real-robot confirmation. Hardware permitting, the same three tasks are run against CFG-ADDA at matched data. Success rate is the metric, with confidence intervals from repeated rollouts.

LiDAR segmentation. We run SynLiDAR to SemanticKITTI and SynLiDAR to nuScenes at 1%, 10% and full labeled-real fractions, with mIoU as the metric. Here we report no reported result found on Lacuna for the low-label fractions specifically, so all baselines are rerun.

Image classification. VisDA-2017 (synthetic renderings to real photos) in a few-labeled-target variant serves as a third, purely visual check. The file reports Tighter-VR at 78.9% in the unsupervised setting, but no reported result found on Lacuna for a few-label VisDA-C setting.

Falsification condition. The proposal fails if per-block weights never beat the best grid-searched constant and measured bias predicts transfer no better than feature distance.