Robots are often trained entirely in simulation and then deployed on real hardware with no further training. To make this work, practitioners use domain randomization: during training they randomly vary physical and visual properties such as mass, friction, latency, textures and lighting, so that the real world looks like just one more variation. How well such a policy transfers depends on what it saw in training. It also depends on the shape of the neural network that implements it, for example whether it has memory, how deep its image encoder is, and how it normalizes activations. Yet the architecture of a sim-to-real policy is almost always chosen once, by hand, and justified by one or two real-robot runs. Computer vision learned to compare whole families of architectures statistically rather than through lucky single models, and found that rankings measured on cheap proxy tasks often carry over to harder ones. Robotics has no equivalent evidence about whether architecture rankings measured in simulation carry over to the real robot.
This note proposes a design-space study of robot policy architectures across the reality gap. It asks two things. First, when families of architectures are compared under domain randomization in simulation, is their ordering preserved on physical hardware? Second, can a cheap simulation-only test predict the real-world ordering? The candidate test evaluates policies on simulated conditions deliberately placed outside the randomization range. The hypothesis is that ordinary in-distribution validation reward ranks architectures poorly, while out-of-range simulated shifts rank them well. If that holds, architecture selection for sim-to-real becomes a cheap, offline procedure rather than a hundred-hour hardware campaign.
Core Claim
We claim that the transferability of a policy architecture across the reality gap is a property of its design space. A design space here means a parameterized family of networks, sampled at matched size. We further claim that this property can be measured in simulation, provided the measurement probes the regime that domain randomization theory does not cover: real dynamics that lie outside the randomized range. Concretely, we expect the rank correlation between a design space's real-robot success and its simulated success under out-of-range perturbations to be substantially higher than its correlation with success inside the training randomization range. We also expect architectural factors beyond the coarse choice of memory versus no memory to explain a measurable share of real-world variance once memory is controlled for. Both expectations can fail, and the study is designed so that either failure is informative.
Where This Sits in Prior Work
Three papers motivate the question by showing that architecture matters for sim-to-real. None of them studies it systematically. Sim-to-Real Transfer of Robotic Control with Dynamics Randomization trained a pushing policy on a Fetch arm and compared three hand-picked networks on the real robot. An LSTM (a recurrent network that carries memory across timesteps) reached 0.89 success, a feedforward network with stacked history reached 0.67, and a feedforward network without randomization reached 0.51. These are single point estimates on one task. Understanding Domain Randomization for Sim-to-real Transfer proved that memory is necessary for randomized policies to identify the system they are in implicitly. The proof assumes the real world lies inside the randomized distribution, and it says nothing about structure beyond the memory class. Asymmetric Actor Critic for Image-Based Robot Learning showed that splitting information between an image-only actor and a privileged-state critic changes what transfers, but it used fixed networks. We take the out-of-support regime that the theory excludes and turn it into the central measurement. We also replace point estimates with distributions.
Two lines of work supply our methods. On Network Design Spaces for Visual Recognition compared architecture families through empirical distribution functions of error at controlled complexity, using Kolmogorov-Smirnov tests. It warned that single models conflate family quality with seed noise. NAS-Bench-201 showed that rankings of 15,625 cell architectures are stable across CIFAR-10, CIFAR-100 and ImageNet-16-120. We reuse the distributional methodology, but our evaluation is policy success across a physical gap rather than accuracy across image datasets.
NAS-Bench-201 also supplies a constraint. It found that the way batch normalization computes its statistics can reorder architecture rankings. Simulation-to-real shift changes input statistics in exactly the way that would trigger this, so we treat the normalization scheme as an explicit factor in the design space and do not leave it as a hidden default.
The closest connecting work does something different from what we propose. Adapting Neural Architectures Between Domains searches for domain-invariant topologies with an adversarial domain discriminator, but only for image classification and not for control or randomized physics. Auto-Agent-Distiller and EMNAS-RL search architectures for reinforcement learning in fixed simulators, without randomization or hardware. Auto-Agent-Distiller did find that larger ResNets degrade reinforcement learning, which warns us that capacity must be matched. Validate on Sim, Detect on Real ranks 60 randomization-trained policies by combining simulation validation with out-of-distribution detection on real images, but its architecture is fixed. It is the strongest existing selection method and the main baseline below. We deliberately do not run architecture search: before searching, one has to know whether a simulation signal predicts real outcomes, and that is the question we study.
Research Program
We build a policy design space with six factors. The first is memory type: none, stacked history, GRU, LSTM, or a small causal transformer. The second is encoder depth and width. The third is whether residual connections are used. The fourth is information gating on the history input, in the style of the entropy-penalized attention masks of Priors, Hierarchy, and Information Asymmetry for Skill Transfer in Reinforcement Learning. The fifth is normalization: batch normalization with running statistics, layer normalization, or none. The sixth is actor-critic information asymmetry. We sample roughly 200 configurations within a fixed parameter band. Every configuration is trained with the same reinforcement learning algorithm, the same dynamics and visual randomization ranges, three seeds, and a small fixed tuning budget spent identically on learning rate and entropy coefficient. The fixed tuning budget addresses NAS-Bench-201's warning that a single shared hyperparameter setting can penalize some architectures unfairly.
Each trained policy is then scored on four test conditions. The first is the training randomization distribution. The second is held-out parameter values inside that range. The third is out-of-support shifts: friction, mass and latency pushed beyond the training bounds, plus effects that were never randomized at all, such as actuator backlash, an attached unmodeled mass in the spirit of Peng et al.'s chip-packet test, and camera noise models not seen in training. The fourth is the real robot, for a stratified subset of policies.
For each design factor we compute empirical distribution functions of success under each condition. Kolmogorov-Smirnov statistics then tell us whether one family dominates another. Kendall's $\tau$ between the simulation-condition ranking and the real-robot ranking, taken over design-space cells, is the primary quantity. A variance decomposition then asks how much real-world variance architecture explains once the memory class is fixed, compared with seed variance.
The tasks are chosen to span the precision spectrum where transfer is known to break. We use ManiSkill2 PickCube, for which ManiSkill2 reports 91% success in simulation and 60% on hardware. We use robosuite Lift on a Franka Panda, matching the setup of Validate on Sim, Detect on Real and robosuite. We also use a Fetch-style puck push to connect directly to Peng et al. The real-robot budget follows Validate on Sim, Detect on Real, which needed 130 hours for 60 policies. We plan about 48 real policies: two per design-space cell, chosen to span the simulated rankings so that the correlation estimate is not dominated by easy extremes.
Baselines
The naive reference point ranks architectures by the success of each family inside the training randomization distribution. This is what practitioners implicitly do, and it corresponds to the simulation-validation term of Validate on Sim, Detect on Real. It uses the same trained policies as every other method, so the only thing that differs is the ranking signal, and the comparison is exactly fair. Our out-of-support proxy beats it if its Kendall $\tau$ with real-robot rankings is higher, with a bootstrap confidence interval over design-space cells that excludes zero difference.
The strongest existing selection method is Validate on Sim, Detect on Real (VSDR) itself. It fits a Gaussian mixture model to internal activations on a small set of real images and combines the resulting out-of-distribution score with simulation validation. Its authors report that it "improved policy ranking accuracy in 87% of cases compared to using simulation validation alone" on robosuite Lift. We rerun it on our policy pool with the same real images given to every method. Because it uses real data and our proxy does not, matching it counts as a win and exceeding it counts as a strong win. Its activation-based score is also not obviously comparable across different architectures, which is part of what we test. We additionally report the combination of the two, since out-of-support simulation tests and real-image outlier detection may capture different failures.
The strongest existing architecture point is Peng et al.'s LSTM with dynamics randomization, which achieved 0.89 ± 0.06 real success on Fetch pushing. In our study it is simply one cell of the design space. A design space beats it if its median real success exceeds the LSTM cell's distribution at matched parameters, so the comparison is between families rather than between one tuned model and another. For PickCube the reference figure is ManiSkill2's 60% real success with its standard network.
Two ablations isolate the key ingredient. The first removes the out-of-support condition and keeps only in-support held-out parameters. If it predicts real rankings equally well, the gain comes from held-out evaluation in general, not from leaving the support. The second shuffles which unmodeled effect is probed, to check that predictiveness does not hinge on one lucky perturbation such as latency.
Why not just…?
Why not just use an LSTM, widen the randomization, and stop thinking about architecture? This is the most serious threat. Chen et al.'s theory and Peng et al.'s results both say memory is what matters, and the remaining factors might add nothing. Our variance decomposition tests this directly. If, within a fixed memory class, architecture explains no more real-world variance than seeds do, we will report that architecture beyond memory class is irrelevant for transfer, which is a useful negative result in its own right. Widening the ranges is also not free. The motivating literature describes over-conservative behaviour under broad randomization, and Auto-Tuned Sim-to-Real Transfer reports 0% real success for its randomization baselines on Cabinet Slide and Rope Peg-in-Hole. We run the LSTM cell at two randomization widths so the trade-off is measured rather than assumed.
Why not just select with VSDR and skip architecture study? VSDR chooses among already-trained policies and needs real images. It does not tell a designer which family to train in the first place. Its activation statistics are computed inside each network, so comparing them across different architectures may be confounded by the architecture itself. Running it on our heterogeneous pool will show whether this confound appears. If VSDR ranks architectures as well as our proxy does, we will say so and recommend it.
Why not just fix the simulator with system identification instead? Grounding methods such as Reinforced Grounded Action Transformation and the parameter search in Auto-Tuned Sim-to-Real Transfer reduce the gap, but they require real rollouts for each task and a good starting guess. These methods are complementary to ours, not competing. Even with a better-matched simulator, effects that were never modeled remain, and that residual out-of-support regime is where we expect architecture to matter. As a secondary check we repeat the PickCube ranking with simulator parameters tuned toward the real robot, to see whether architecture rankings remain stable once the gap shrinks.
Why not just stack history frames into an MLP? That is one cell of the design space. Peng et al. already show stacked history at 0.67 against 0.89 for the LSTM on one task. We test whether that gap holds as a distributional result across tasks and sizes.
Decisive Experiments
The study hinges on three checks. The first is whether real-robot design-space rankings agree with in-distribution simulated rankings, measured by Kendall $\tau$ over cells on PickCube, Lift and Push. A high $\tau$ would mean NAS-Bench-201-style stability survives the reality gap and in-distribution simulation is a good enough proxy. The second is whether the out-of-support proxy raises $\tau$ over both the naive ranking and VSDR, with the in-support-only ablation ruling out the trivial explanation. The third is whether non-memory factors, especially normalization scheme and history gating, shift the real-success distributions at matched size, as measured by Kolmogorov-Smirnov statistics with seeds controlled.
A positive outcome on the first two checks would justify the next project: architecture search for sim-to-real driven entirely by out-of-support simulated evaluation. A negative outcome on the third would settle that memory class is the only architectural decision worth making for zero-shot transfer.
References
- Sim-to-Real Transfer of Robotic Control with Dynamics Randomization
- Understanding Domain Randomization for Sim-to-real Transfer
- Asymmetric Actor Critic for Image-Based Robot Learning
- On Network Design Spaces for Visual Recognition
- NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search
- Adapting Neural Architectures Between Domains
- Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search
- Robust Evolutionary Multi-Objective Neural Architecture Search for Reinforcement Learning (EMNAS-RL)
- Validate on Sim, Detect on Real - Model Selection for Domain Randomization
- Priors, Hierarchy, and Information Asymmetry for Skill Transfer in Reinforcement Learning
- ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills
- robosuite: A Modular Simulation Framework and Benchmark for Robot Learning
- Auto-Tuned Sim-to-Real Transfer
- Reinforced Grounded Action Transformation for Sim-to-Real Transfer