Rehearsing the Reality Gap: Calibrating Ensemble Disagreement on Held-Out Randomization Factors

Robots and self-driving systems trained entirely in simulation are usually made robust by domain randomization: randomly varying weather, textures, road layouts and physics during training so that the real world looks like one more random variation. What domain randomization does not provide is a warning. Nothing tells you, at deployment time, that the camera is now seeing something the training variations never covered. The natural candidate for that warning is a deep ensemble, several networks trained from different random starts, whose disagreement rises on unfamiliar inputs. Ensembles, however, are least trustworthy exactly where they are needed. On real distribution shifts, a Gaussian ensemble that covers the truth 90% of the time on familiar data drops to roughly 59% coverage, and no tested method keeps its nominal coverage (How Reliable is Your Regression Model's Uncertainty Under Real-World Distribution Shifts?).

This note proposes using the structure of domain randomization itself to fix that miscalibration without any real data. The randomization is already split into factor families: weather, surface appearance, road topology, traffic behavior, sensor placement. We repeatedly train the ensemble with one family held out entirely. We then measure how disagreement relates to actual error and to driving failure when the system meets the missing variation, and fit a single mapping from disagreement to risk that has been "rehearsed" on many out-of-support shifts. At deployment, the ensemble is trained on all factors and uses this rehearsed mapping. When predicted risk crosses a threshold, a conservative fallback behavior takes over. The testable claim is that shifts which are outside the randomization support in simulation are a good stand-in for the unknown shift to reality, at least in how they distort ensemble confidence.

Core Claim

Ensemble disagreement becomes a usable zero-shot readiness signal once it is calibrated against deliberately withheld randomization factors instead of in-distribution validation data. Calibrating on in-distribution data is what produces the collapse from 90% to about 59% coverage. Calibrating on held-out factors should produce risk estimates that stay roughly calibrated on genuinely unseen conditions: unseen towns and weather in CARLA, a sensor change that appears in no family, and real KITTI images. In closed loop, gating on that risk should reduce infractions more than gating on raw disagreement.

This is aimed at a specific weakness in the theory. Understanding Domain Randomization for Sim-to-real Transfer models domain randomization as a decision problem with a hidden environment parameter. Its guarantees assume two things. First, reality lies inside the randomized support. Second, the agent can always recover from mistakes; if it cannot, the gap grows linearly with the task horizon. Collisions and road departures are unrecoverable states, and reality is rarely inside the support. A calibrated per-frame warning that triggers a safe primitive before such states are reached is a practical response to both violated assumptions. The theory motivates the design but does not certify it.

Relation to Prior Work

The closest work is Safe Domain Randomization via Uncertainty-Aware Out-of-Distribution Detection and Policy Adaptation. That paper trains critic ensembles with a diversity loss that forces disagreement on randomized "repulsive" transitions. It expands the randomization range when disagreement on a small real target-domain dataset exceeds a threshold, and it validates on an ANYmal-D quadruped. That work supplies the ensemble-as-gatekeeper idea, but it needs real target data and uses a raw threshold. We need no target data, and our contribution is the calibration protocol, not a new ensemble loss. Their diversity loss therefore enters our study as a competitor.

Three curriculum methods decide where to randomize using signals other than uncertainty:

All three shape training. None produces a deployment-time estimate of risk, so they are orthogonal to our question. We keep the training randomization fixed so that the monitor is the only thing that varies.

On the perception side, three papers constrain or motivate the design:

Research Program

The system is a modular driving stack in the style of Sim2Seg, built in CARLA: An Open Urban Driving Simulator. A perception ensemble of $M=5$ networks maps camera images to semantic segmentation, depth, and 3D boxes. A policy consumes these canonical maps and chooses among a small set of driving primitives: follow lane, turn at junction, yield, and a slow-and-stop fallback. Each primitive outputs waypoints to a low-level controller. Primitives follow Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks, where primitives transferred zero-shot to a real arm, because they give the monitor a meaningful action to trigger.

Randomization is organized into $K$ factor families, each drawn from its own ranges:

  • weather and lighting;
  • surface textures and vehicle appearance;
  • town topology, using disjoint CARLA towns;
  • traffic density and aggressiveness;
  • camera pose and noise.

The calibration protocol works as follows. For each family $k$, we train an ensemble on all families except $k$ and run it, together with the policy, on episodes where family $k$ is varied beyond anything that ensemble saw. For each frame we record ensemble disagreement $d$, defined as the mean per-pixel variance of depth plus the disagreement in segmentation labels, and the realized outcome $y$, which is whether an infraction occurs within the next two seconds. Pooling across all $K$ held-out runs, we fit a monotone map $r(d) \approx P(y=1 \mid d)$ by isotonic regression. For interval tasks such as depth, we instead fit a rescaling factor on the pooled held-out data so that nominal 90% intervals achieve 90% coverage.

The deployed ensemble is trained on all families and uses $r$ unchanged. The fallback primitive is triggered when $r(d)$ exceeds a threshold $\tau$, chosen in simulation to trade intervention rate against infractions.

A secondary ingredient, which we ablate, is factor-partitioned members. Each deployed member is trained with a different random subset of families narrowed, so that disagreement is diverse by construction and not only by random seed.

Baselines

The naive reference is the same stack with no monitor, trained with uniform randomization over all families. It establishes the infraction rate that any gate must reduce. It also carries a known risk: CARLA's own study reports navigation success falling from 80% to 24% for a modular pipeline, and from 86% to 40% for imitation learning, when moving from the training town to an unseen town (CARLA).

The standard uncertainty baseline is a seed-diverse deep ensemble calibrated on in-distribution held-out frames and gated on the same fallback. This is the configuration whose coverage How Reliable is Your Regression Model's Uncertainty Under Real-World Distribution Shifts? reports dropping to about 59% on Cells-Tails and to 55–70% on HistologyNucleiPixels. We expect a similar collapse here. Beating it means held-out-coverage within 5 points of nominal where this baseline is 20 or more points below.

The strongest existing competitor is the diverse-critic-ensemble approach of Safe Domain Randomization via Uncertainty-Aware Out-of-Distribution Detection and Policy Adaptation. We adapt it to be sim-only by using held-out-family transitions as its repulsive data, which removes its need for real target data, and use its critic variance as the gate. That paper reports matching or exceeding baselines in 84% of comparisons on locomotion. I found no reported result on Lacuna for failure-prediction AUROC of any ensemble monitor in closed-loop urban driving, so the bar is set by our own reproduction.

A single-network MC-dropout monitor with the same calibration protocol tests whether the ensemble matters at all.

For fairness, all monitors share:

  • the perception architecture and total parameter budget, so dropout gets one wider network;
  • the randomized training frames, with an equal number of frames per method including the held-out-family runs;
  • the policy;
  • the fallback primitive;
  • threshold tuning on the same simulation validation towns.

Beating the strongest competitor means higher failure-prediction AUROC and lower calibration error on the unseen evaluation conditions, and a better driving score at an equal fallback rate.

Two ablations isolate the key ingredients. The first is a two-by-two design: seed-diverse versus factor-partitioned members, crossed with in-distribution versus held-out-family calibration. This separates the contribution of calibration from that of member construction. The second varies the number of held-out families used for calibration from 1 to $K$. If calibration quality does not improve with more rehearsed shifts, the claimed mechanism is doubtful.

Why not just…?

Why not just test the policy on a held-out town and report the success rate? That gives one number per deployment condition, not a per-frame warning. It cannot trigger a fallback before an unrecoverable collision, which is what the recoverability assumption in Understanding Domain Randomization for Sim-to-real Transfer calls for. For a pure go/no-go decision, however, an aggregate held-out score may be enough. We therefore report it as a separate readiness predictor and check whether per-frame gating improves driving score beyond what the aggregate prediction implies.

Why not just use conformal prediction, which has guarantees? Its guarantees require calibration and test data to be exchangeable, and real shifts break that assumption. The regression shift benchmark shows conformal intervals losing coverage under real shift. This is a real threat in one respect: our held-out-family calibration is itself a conformal-style recalibration on shifted data. The two-by-two ablation includes split conformal calibrated on held-out families. If it matches our isotonic map, the contribution is the calibration data, not the fitting method, and we will say so.

Why not just use MC dropout, since five networks are too slow? What Are Bayesian Neural Network Posteriors Really Like? shows dropout samples near one solution and underrepresents posterior diversity, which predicts weaker disagreement on novel inputs. Latency is a real cost, so we report frames per second for both monitors. If dropout with held-out calibration comes within noise of the ensemble, we recommend dropout.

Why not just randomize more widely and skip the monitor? Uniform randomization wastes training on mastered settings and yields conservative policies (Active Domain Randomization). More importantly, no range covers factors nobody thought to randomize. Our "never-randomized" sensor condition, described below, directly measures what wider randomization cannot address.

Proposed Experiments and Decisive Checks

Training uses CARLA towns 1, 3, 4 and 6 with the five families above. The simulated surrogate reality has three parts. The first is two towns never used, one being CARLA Town 2, which has a documented generalization gap. The second is weather combinations outside every sampled range. The third is a camera height and field-of-view change that belongs to no family, so it tests truly unrehearsed shift.

Metrics are:

  • AUROC for predicting infractions two seconds ahead;
  • expected calibration error of $r(d)$;
  • 90% depth-interval coverage;
  • closed-loop Driving Score, Route Completion and Infraction Score at matched fallback rates.

For context, a strong fully engineered CARLA stack reaches a Driving Score of 76.18 (InterFuser) and TCP-Ens reaches 75.137 (TCP). Those were not trained under our zero-shot protocol, so they are not direct comparisons.

The real-data check runs the CARLA-trained 3D-box ensemble on KITTI and correlates box variance with localization error. The in-domain KITTI-trained ensemble of Real-Time Evaluation of Perception Uncertainty and Validity Verification of Autonomous Driving reached 0.657 orientation correlation and serves as a ceiling. The decisive question is whether held-out calibration brings real-KITTI coverage close to nominal where in-distribution calibration does not.

The idea fails if either of two things happens. The first is that calibration fitted on held-out families is no better than in-distribution calibration on the never-randomized camera condition and on KITTI; that would mean rehearsed shifts do not predict unrehearsed ones. The second is that the sim-only adaptation of the diverse-critic baseline matches our gate. If the Scaled Smart City testbed used by Zero-Shot Autonomous Vehicle Policy Transfer is available, a small physical run comparing fallback-triggered collisions would be the final check. It is not required for the main conclusions.