Agreement Across Randomizations as a Label-Free Predictor of Sim-to-Real Transfer

Robots trained in simulation often fail on real hardware, and the standard fix is domain randomization: training under heavily perturbed textures, lighting, masses and frictions so that the real world looks like one more variation. Nobody can yet tell, before running real trials, which trained policy will survive the jump. This proposal borrows an idea from neuroscience-inspired AI. In functional convergence, networks trained on the same task under the same physical constraints end up with similar internal representations, much as brains do. We turn that observation into a test. Train many vision-based policies under different randomization schemes, then show all of them the same small set of unlabeled real camera images. The parts of the representation they agree on are what the task itself dictates. The parts where they disagree are shortcuts tied to a particular simulator. We hypothesize that a policy's agreement with the group on real images, compared with its agreement on simulated images, predicts its real-world success. We further hypothesize that pushing an encoder toward the shared subspace improves transfer.

The opportunity is practical. The best existing transfer predictor, full-state probing, regresses simulator ground truth (object poses, joint angles, lighting) from frozen features. It correlates with control performance at $r = 0.753$ (Capturing Visual Environment Structure Correlates with Control Performance), but it needs state labels, and those labels exist only in simulation. It therefore measures how well a representation encodes the simulator, not how it behaves on real pixels. A convergence score needs only a few hundred unlabeled real frames and no policy rollouts.

Core Claim

Representational agreement among independently randomized policies, measured on unlabeled real images with a statistically corrected similarity measure, predicts zero-shot real-world success. The prediction should hold beyond what in-simulation success and simulator-based state probing already explain. The shared subspace is also usable as a training target, which a score alone is not.

The closest work does not test this. Capturing Visual Environment Structure Correlates with Control Performance supplies the evaluation protocol and the strongest competitor, but it never compares models with each other on real images. The convergence literature has looked only at passive image viewing, comparing hundreds of vision and language models against human fMRI (Alignment between Brains and AI) or aligning a classifier with macaque inferior temporal (IT) cortex (Aligning Model and Macaque IT Representations). Neither line has studied closed-loop control or randomization. Better artificial intelligence does not mean better models of biology shows that very capable models drift from biology by exploiting textures and backgrounds. That finding motivates the proposal: randomization-specific shortcuts are the robotic analogue of those shortcuts, and agreement across randomizations is a way to filter them out.

The Measurement

Let $f_i$ be the encoder of policy $i$, and let $X_{\text{real}}$ and $X_{\text{sim}}$ be $N$ matched real and simulated frames from the same task. Here $N$ is a few hundred, and matching is by scene layout, not pixels. We define

$$ R_i = \frac{\frac{1}{|P|}\sum_{j \in P,\, j\neq i} \mathrm{dCKA}\big(f_i(X_{\text{real}}), f_j(X_{\text{real}})\big)}{\frac{1}{|P|}\sum_{j \in P,\, j\neq i} \mathrm{dCKA}\big(f_i(X_{\text{sim}}), f_j(X_{\text{sim}})\big)} $$

The symbols are as follows. $P$ is a reference pool of policies trained under deliberately different randomization schemes. dCKA is the debiased form of centered kernel alignment, a similarity score between two sets of features computed over the same inputs. The ratio asks how much of a policy's agreement with its peers survives the move to real pixels.

We use the debiased form because of a constraint from Correcting Biased Centered Kernel Alignment Measures. That paper shows standard CKA approaches 1 on random matrices when feature dimension is large relative to sample count. Our regime, with high-dimensional backbones and a few hundred real frames, is exactly the one where biased CKA would produce false agreement.

Research Program

We would build a population of about 60 image-based policies on Meta-World (Meta-World) and RoboCasa tasks, with SimplerEnv's real-matched environments as an intermediate check. The population crosses four factors. The first is the visual backbone: frozen CLIP, frozen DINOv2, an ImageNet ResNet, or trained from scratch. The second is the randomization scheme: none, texture-and-lighting only, dynamics only, uniform randomization at two widths, and active randomization (Active Domain Randomization). The third is training with or without a privileged-state critic, following Asymmetric Actor Critic. The fourth is random seed.

We would then deploy a subset of about 20 policies on a physical tabletop arm comparable to the Xarm6 used by the full-state paper, on pick, push and drawer tasks. Each policy gets a fixed number of trials per task, and we record success rates.

The second half of the program is an intervention. We would compute the top principal directions shared across the pool on mixed sim-and-real frames. A new policy is then trained with an auxiliary loss that keeps its encoder close to that consensus subspace, using only unlabeled real frames. This tests whether convergence carries causal weight or is only a correlate.

A secondary analysis connects back to neuroscience. We would score the same encoders against macaque IT responses on the HVM image set and against human fMRI from the Natural Scenes Dataset, again using debiased CKA. The question is whether randomization width moves alignment with the ventral visual stream, the brain pathway that recognizes objects. We expect this analysis to show dissociation, not agreement. IT cortex discards pose information, and control requires it.

Baselines

The naive reference is in-simulation success rate as the predictor of real success. If the convergence score adds nothing over it, the idea fails. We report partial correlations of every predictor controlling for sim success.

A cheap but weak reference is ImageNet linear probing on the frozen encoder. Capturing Visual Environment Structure Correlates with Control Performance reports it correlates with control at $r = -0.020$, and semantic segmentation at $r = -0.042$. We rerun it on our population to confirm that it fails here too.

The strongest competitor is full-state probing from the same paper: an MLP regressing simulator state from frozen features, with a reported Pearson $r = 0.753$ against control performance. It is strongest because it directly measures what control needs. We run it with the authors' protocol on the identical policy population, with the same simulated data volume used for our sim frames and the same small probe-tuning budget. Its correlation with real success is the number to beat. Beating it means the convergence score yields a higher Spearman correlation with real success, together with a lower mean maximum rank violation (the ranking-error measure used in that paper), with bootstrap confidence intervals over policies. A complementary win also counts: if the two predictors combined explain substantially more variance than full-state probing alone, the label-free signal is additive.

A brain-alignment baseline scores each encoder by debiased CKA with macaque IT on HVM. This tests the naive version of the cross-field idea, that brain-like means transferable.

Two ablations isolate the key ingredient. The first restricts the pool to policies sharing one randomization scheme. If diversity of randomization is what exposes shortcuts, predictive power should drop. The second replaces debiased CKA with biased CKA. We expect inflated, uninformative scores at our sample sizes.

Why not just…?

Why not just use full-state probing? It is the real threat, and it may win in simulation-only evaluation. Its labels are simulator state, however, so it cannot see how an encoder reacts to real pixels. Two encoders can decode simulated poses equally well and still differ sharply on real lighting. The deciding experiment is the partial correlation of our score with real success after controlling for full-state accuracy. If that partial correlation is zero, we report that convergence offers nothing beyond probing apart from being label-free.

Why not just measure action disagreement of an ensemble on real images? This is cheaper and plausibly about as predictive, so we run it with the same pool and frames. It cannot say which features are shared, and so cannot supply the consensus-subspace training signal. If it matches us at prediction, the paper's contribution narrows to the intervention, and we would say so.

Why not just measure how far each encoder's real features sit from its sim features? Single-model invariance rewards collapse: a constant encoder is perfectly invariant. Normalizing real-image agreement by sim agreement, and pooling across distinct randomizations, guards against this. We include the invariance score as a baseline to show the difference empirically.

Why not just use brain alignment directly? Aligning Model and Macaque IT Representations found that IT-alignment gains vanished on held-out categories. Ventral-stream representations also throw away the metric pose that manipulation needs. We test brain alignment as a baseline and expect it to underperform. A null result here is informative for the convergence literature, because it locates where passive-vision convergence stops being relevant to embodied control.

Decisive Checks

The proposal succeeds if three things hold. First, $R_i$ predicts real-arm success above in-sim success, with a significant partial correlation. Second, $R_i$ adds explained variance beyond full-state probing. Third, consensus-subspace training raises real success over a matched policy without that loss, trained with the same compute and randomization. It fails cleanly if any of the following occur: the same-scheme pool ablation predicts as well as the diverse pool, biased and debiased CKA give the same rankings (which would mean the measurement is not doing what we claim), or the intervention leaves real success unchanged.