Where Does the Ensemble Disagree? Simulator-Grounded Sanity Checks for Uncertainty Maps in Sim-to-Real Policies

Robots trained in simulation are often sent into the real world with a warning light attached: several copies of the policy are trained from different random starting points, and when they disagree about what to do, the robot is probably seeing something it was never trained on. That light only says that something is wrong, not what. This proposal asks for a map of where the disagreement comes from in the camera image, for example the unfamiliar table texture, the shifted camera, or the new object. It also asks whether such a map can be trusted. Interpretability research has repeatedly shown that pixel heatmaps can look convincing while ignoring the model entirely. The opportunity is that simulation removes the usual obstacle to checking them. In a simulator we can plant an unfamiliar factor at a known place in the scene, so we know where the disagreement map should point. We can also re-render that region with familiar content, so we can test whether removing it actually removes the disagreement.

The core claim is that domain randomization, the practice of training in simulators whose textures, lighting, camera pose and physics are randomly varied so that the real world looks like one more variation, supplies the ground truth that explanation sanity checks have lacked. With that ground truth we can build positive faithfulness tests for maps of ensemble disagreement on visuomotor policies, meaning policies that map camera images to motor actions. Until now the field has had only negative tests that catch broken methods. We expect two findings. First, several popular attribution methods will produce disagreement maps that pass existing randomization tests and still fail to locate planted shifts, especially shifts away from the image center. Second, disagreement maps that do pass can tell the training loop which randomization range to widen, which a single scalar variance cannot do.

What exists and what is missing

Three bodies of work frame the problem. On the interpretability side, Sanity Checks for Saliency Maps introduced the model parameter randomization test, which scrambles a network's weights and checks whether the heatmap changes. It found that Guided Backpropagation and Guided Grad-CAM barely change, while vanilla gradients and Grad-CAM degrade to noise. When Explanations Lie explained the failure. Many modified backpropagation rules multiply non-negative matrices, and that product collapses to a fixed, input-driven direction, with cosine-similarity convergence above 0.99 on VGG-16 and ResNet-50. When Random Saliency Looks Trained added a confound. Zero-padded CNNs give center-heavy saliency even at random initialization, with center-bias AUC above 0.90 against about 0.50 for Vision Transformers. As a result, trained and random maps agree on centered images (about 0.40 correlation) but not when the object sits in a corner (0.15). These papers constrain the design: any method we propose must survive weight scrambling, and every test must place shifts off-center.

Two papers supply methods we reuse and motivate the ground-truth idea. Sanity Simulations for Saliency Methods used synthetic images with known model logic and found that all tested methods fail on multi-object reasoning and degrade with natural backgrounds. We borrow its logic of controlled ground truth but obtain it from a physics simulator rather than a toy dataset. A Benchmark for Interpretability Methods in Deep Neural Networks (ROAR) showed that deleting pixels from images creates distribution-shift artifacts, so it retrains models from scratch. Under that protocol, gradient and Integrated Gradients do no better than random removal, while SmoothGrad-Squared drives ImageNet accuracy to about 29% at 90% removal against about 63.5% for random removal. A simulator lets us skip retraining: we re-render the highlighted region with in-distribution randomized content instead of masking it.

The two closest bridges each cover only half of the problem. Sanity Checks for Explanation Uncertainty measured how much ensemble members' explanations vary and found that deep ensembles pass weight and data randomization checks on CIFAR-10 and California Housing. That work quantifies the uncertainty of an explanation. We instead seek an explanation of uncertainty: an attribution of the ensemble's disagreement itself. It also reports that Guided Backpropagation passes, which is in tension with the two papers above that show Guided Backpropagation is insensitive to weights. Our null-ensemble control is designed to resolve that tension. Safe Domain Randomization via Uncertainty-Aware OOD Detection (UARL) uses variance across an ensemble of value-estimating critics to gate deployment and to widen randomization ranges progressively. It was validated on MuJoCo and an ANYmal-D quadruped. Its signal is a single scalar. It treats the networks as black boxes, and it widens ranges without knowing which factor caused the alarm. UARL is the strongest system-level competitor we must beat.

Research Program

We train an ensemble of $M=5$ image-based actors, each with an asymmetric critic in the style of Asymmetric Actor Critic for Image-Based Robot Learning. In that design the critic sees privileged simulator state and the actor sees randomized RGB images. Because only the actors see pixels, we define disagreement on actions:

$$D(o) = \frac{1}{M}\sum_{m=1}^{M} \lVert \pi_m(o) - \bar\pi(o) \rVert^2,$$

where $o$ is the camera image, $\pi_m(o)$ is member $m$'s action, and $\bar\pi(o)$ is the ensemble mean. A disagreement map is any attribution of the scalar $D(o)$ to pixels. We compare five ways to compute it: vanilla gradient, Integrated Gradients, SmoothGrad-Squared, occlusion (re-measuring $D$ after graying patches), and simulator re-rendering occlusion (re-measuring $D$ after re-rendering a patch's scene content from the training randomization distribution). We include Guided Backpropagation as a planned negative control. We also run all of these on two backbones: a zero-padded CNN and a ViT, plus a CNN with circular padding. This lets us separate center bias from real signal.

The evaluation protocol has three tests. The planted-shift localization test renders scenes in which exactly one factor leaves its training range at a known place. Planted shifts include an out-of-range texture on the target object, on a distractor, or on the background; a lighting change confined to one region; and a novel object geometry. Each shift appears at the center, the edges and the corners. The simulator's segmentation mask gives the ground-truth region. The counterfactual re-render test is a simulator analogue of ROAR. We restore the top-$k\%$ attributed pixels' underlying scene entities to in-distribution randomizations and measure how much $D$ falls, compared with restoring a random region of equal area. The randomization tests are the classical checks adapted to ensembles. We cascade weight scrambling across all members. We train a behavior-cloned ensemble on shuffled observation–action pairs. Most importantly, we add a null ensemble: five randomly initialized networks whose disagreement maps should not localize anything. If a method's maps look the same for trained and null ensembles, the method fails, regardless of whether its measured "uncertainty" rises under randomization.

The payoff experiment uses passing disagreement maps to guide range expansion. When target images trigger high $D$, we associate the attributed regions with scene entities. We then widen only the randomization ranges of factors attached to those entities, retrain, and compare against UARL-style uniform widening.

Baselines

The naive reference points are a uniform random heatmap and a fixed centered Gaussian heatmap. The centered Gaussian matters because When Random Saliency Looks Trained shows that center bias alone can mimic trained saliency. Any method must clearly beat both on localization AUC, the area under the curve for separating planted-region pixels from other pixels, and on the fraction of attribution mass inside the planted region, measured separately for corner shifts and center shifts. On range expansion, the do-nothing reference is the fixed, hand-set randomization ranges with scalar disagreement used only as an alarm.

The strongest existing system is UARL (Safe Domain Randomization via Uncertainty-Aware OOD Detection). Its file entry reports only that UARL matched or exceeded baselines "in 84% of benchmark comparisons" on MuJoCo. We found no reported number on Lacuna for UARL on image-based manipulation, so we will reimplement its scalar-variance gatekeeper and uniform widening on our tasks. The reimplementation will use the same ensemble size, the same target proxy images, the same total simulator steps and the same tuning budget for thresholds. Beating UARL means reaching higher target-domain success after the same number of widening rounds, or the same success with fewer rounds.

The strongest explanation-side competitor is explanation-uncertainty variance from Sanity Checks for Explanation Uncertainty: the per-pixel variance of the members' individual action saliency maps. We run it with identical members and attribution methods. Beating it means better planted-shift localization and a larger drop in disagreement under counterfactual re-rendering.

As an uncertainty-estimator ablation, we replace the ensemble with an MC Dropout policy, which keeps dropout active at test time and samples one network repeatedly, matched in inference cost per frame. The ablation that isolates our key ingredient is the null ensemble, together with single-member saliency of the mean action. Together they tell us whether localization comes from disagreement or merely from where any policy looks. Finally, we run an oracle that sweeps each randomization factor directly in simulation and measures $D$. This oracle is an upper bound, not a competitor.

Why not just…?

Why not just sweep each randomization parameter in simulation and see which one raises disagreement? This is the most serious threat, and in simulation it is exactly our oracle. Its limitation is at deployment. Real target images arrive without factor labels, and a sweep can only find factors that someone already parameterized, such as an unmodeled reflective surface. The payoff experiment uses only target images, never factor labels, and reports what fraction of the oracle's gain disagreement maps recover. If that fraction is small, we will report that disagreement maps are a diagnostic tool and not a useful tuning signal.

Why not just take the gradient of the disagreement and stop there? The vanilla gradient is one of our candidate estimators, not a strawman. Gradients pass weight randomization in Sanity Checks for Saliency Maps but do no better than random under ROAR in A Benchmark for Interpretability Methods in Deep Neural Networks. So whether gradients are enough is an empirical question, and the planted-shift and re-render tests answer it. The contribution is the protocol that decides which estimator to trust, whichever one wins.

Why not just use the scalar variance, as UARL does? A scalar can gate deployment, but it cannot say which range to widen. The "shotgun" problem described in the UARL paper is that overly wide uniform ranges produce conservative policies. The targeted-versus-uniform widening comparison under equal compute tests this directly. If uniform widening matches targeted widening, the map adds no control value, and we will say so.

Why not just use MC Dropout to avoid training five policies? Epistemic Uncertainty Estimation with Deep Ensembles argues that dropout samples stay near a single solution and underestimate uncertainty under shift. What Are Bayesian Neural Network Posteriors Really Like? finds that ensembles approximate the Bayesian average well. Neither result covers spatial attribution, so we test dropout head-to-head under matched cost.

Proposed Experiments

The primary testbed is ManiSkill2 (ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills) RGB-D tasks, centered on PickCube. That benchmark reports a policy trained only in simulation reaching 91% success in simulation and 60% on a physical robot without fine-tuning. This gives a known sim-to-real gap to diagnose, though our own hardware numbers will be compared against our own reproduced baseline rather than against that figure. Fetch pick and push tasks in the style of the asymmetric actor-critic paper provide a second visual testbed. Unseen YCB and EGAD objects in ManiSkill2 supply novel-geometry shifts.

The first decisive check is whether the null ensemble's maps stay unlocalized while trained-ensemble maps localize corner-placed shifts above the centered-Gaussian baseline. Our prediction is that Guided Backpropagation fails this check, even though it shows rising "uncertainty" under randomization as reported by Sanity Checks for Explanation Uncertainty. That would resolve the Gap 4 contradiction: the dispersion such checks measure need not come from the learned weights. The second check is the counterfactual re-render test. Re-rendering the top 10% attributed region should reduce $D$ far more than re-rendering a random 10% region. The third check is whether the CNN-versus-ViT gap in localization of corner shifts tracks the center-bias AUC gap reported for those architectures. The fourth check is the range-expansion comparison against UARL and the oracle in pseudo-real target domains, where a held-out subset of factors is shifted, followed by a small real PickCube trial.

Every metric for disagreement-map localization is new. We found no reported result on Lacuna for disagreement localization on any robotics benchmark.