Built-In Redundancy for Sim-to-Real Transfer: Pathways Blinded to Different Simulator Cues

Models trained on synthetic images often fail on real ones because they learn to rely on cues the simulator produces but reality does not, such as perfect edges, canonical lighting, or repeated textures. The usual fix is to make one representation invariant to everything that differs between simulator and world. That fix has a cost. Some of the cues it removes are useful, and when an unexpected real-world shift breaks whatever cue the model still relies on, there is nothing left to fall back on. We propose the opposite design. Train several parallel "pathways" on synthetic data. Each pathway is deliberately made blind to a different family of simulator nuisance factors, for example one blind to texture and colour and another blind to object geometry and viewpoint, and each is still required to solve the task alone. At deployment on real data, where we do not know in advance which cue families survived the gap, the pathways vote. Agreement between pathways that rely on different evidence selects which unlabeled real samples to trust when adapting the model on the fly.

The inspiration is equifinality, the observation from organisational and cognitive science that one outcome can be reached by several structurally different routes, and that one route can compensate when another is blocked. Two human examples make the point. Limb-different and two-handed people solve the same physics puzzles equally well, one group through deliberate planning and the other through rapid trial and error (Lifelong learning of cognitive styles for physical problem-solving). Teams in a tower-defence game win either by concentrating defences at chokepoints or by spreading them evenly (CPS-TaskForge). Set-theoretic studies of cities and firms add that a factor which looks like a deficit in one configuration can be compensated by another factor (Institutional Configuration Study of Urban Green Economic Efficiency). These works only catalogue recipes after the fact, in small tables. We take the principle as a design rule for networks: make several sufficient routes on purpose, each blocked from a different shortcut.

Core Claim

Synthetic data carries something real data never has: exact labels for the nuisance factors. The renderer knows the lighting, texture, background, and camera pose of every image. Adversarial debiasing trains an encoder so that an auxiliary classifier, the "adversary," cannot recover a protected attribute from its features. With free nuisance labels, this can be applied cheaply and with full supervision. Our claim is that this supervision is better spent assigning different blindness to different pathways than making one representation blind to all of it. The resulting pathways fail on different real images. Their disagreement therefore flags the confident-but-wrong predictions that break entropy-based test-time adaptation. Their agreement yields cleaner pseudo-labels than any single model's confidence can.

Write the pathways as $k = 1,\dots,K$, with encoder $E_k$, classifier $C_k$, and adversary $D_k$. $D_k$ tries to predict the nuisance group $G_k$, which is a subset of the renderer's parameters. Each pathway is trained with

$$\min_{E_k, C_k}\max_{D_k}\; \mathcal{L}_{task}(C_k(E_k(x)), y) \;-\; \lambda\, \mathcal{L}_{adv}(D_k(E_k(x)), g_k),$$

where $x$ is a synthetic image, $y$ its class, $g_k$ its value of nuisance group $G_k$, and $\lambda$ the adversarial weight. The minimax is implemented with a gradient reversal layer, which flips the sign of the adversary's gradient during backpropagation. The groups $G_k$ are chosen so that no single group is removed from every pathway. Together, the pathways keep all the information, while each one lacks some of it.

Relation to Prior Work

Several papers motivate the gap. Tent adapts a trained model on unlabeled test batches by minimising prediction entropy, updating only normalisation scale and shift parameters. It is evaluated on sim-to-real tasks such as SVHN→MNIST, GTA5→Cityscapes, and VisDA-C. It treats the network as a single route, and its own analysis acknowledges that confident errors get reinforced. That failure is exactly what a shortcut left over from the simulator produces. T3A avoids test-time gradients altogether. It freezes the features and rebuilds classifier prototypes from entropy-filtered test samples. It inherits the same reliance on single-model confidence and assumes the frozen features are clean. The normalisation-statistics methods DUA and MemBN re-estimate feature scaling on the target stream, which fixes coarse corruptions such as fog. Rescaling cannot separate a shortcut from the true signal.

Other work constrains the design. Adversarial debiasing in faces (KANFace) finds that some nuisance attributes are entangled with the task and cannot be removed without losing accuracy. The same study finds that wide embeddings let shortcuts survive erasure. This warns us against blinding one pathway to everything, and it tells us to keep pathway embeddings compact. MABR supplies part of the method: it places adversaries at several layers, not only at the output embedding, so that low-level and high-level biases are both removed. MABR, however, builds one debiased text classifier offline. We reuse the multilayer placement, spread it across pathways, and use it for adaptation on image streams.

Pixel-level translation (PixelDA) closes appearance gaps. It cannot fix geometric mismatch between CAD models and real objects, which is one of the nuisance groups our pathways are built to tolerate. The domain adaptation bound of Ben-David et al. says target error depends on source error, the divergence between domains, and the error of the best joint hypothesis. Our pathways trade some source error each for lower divergence along a chosen axis. Agreement at test time is meant to recover the joint.

Research Program

We would build three things. The first is the training recipe above. It uses $K$ pathways, with $K$ between 2 and 4, on a shared early trunk: pathways branch at the middle of a ResNet-50, and adversaries sit at two depths per pathway. The nuisance groups follow the three-layer anatomy of the sim-to-real gap described in the literature: appearance (texture, colour, lighting), geometry (viewpoint, mesh detail), and context (background, clutter).

The second is a test-time procedure. For each incoming real batch, samples on which a qualified majority of pathways agree go into a support set. That set drives either T3A-style prototype updates for each pathway (no gradients) or Tent-style affine updates restricted to agreed samples (with gradients). Each pathway also receives a running weight equal to its agreement rate with the others on the stream. This lets the ensemble lean on whichever route survived the particular target domain.

The third is a diagnostic borrowed from necessary-condition analysis. At test time we block each pathway in turn and measure accuracy. A pathway whose removal always hurts marks a cue family that is necessary on this target. Pathways whose removal is compensated by the others are substitutable. That yields a readable account of which simulator cues transferred.

The main risk is nuisance labels. The file does not say whether VisDA-C ships per-image rendering parameters. If it does not, we re-render a VisDA-style synthetic set from CAD models with logged parameters. We also test a fallback that uses proxy nuisance labels computed from images: low-frequency colour statistics for appearance, and edge-orientation histograms for geometry. Whether proxies suffice is itself a result.

Baselines

The naive reference point is a source-only ResNet-50 trained on synthetic data and applied to real images unchanged. VisDA reports that an AlexNet drops from 99% on synthetic data to 28% on real COCO images without adaptation. We re-measure this with our backbone, since that figure used a weaker network.

The strongest test-time competitor is Tent, run with its published recipe on the same backbone, batch size, and number of stream passes. Tent reports reducing SVHN→MNIST error from 18.2% to 8.2%, and improving GTA5→Cityscapes IoU from 28.8% to 35.8% (Tent). No reported result was found on Lacuna for Tent's VisDA-C accuracy, so we reproduce it. T3A and DUA are run as gradient-free competitors with their released hyperparameters. For reference only, we note that jointly trained unsupervised adaptation reaches 51.62% with DAN and 92.8% for the challenge winner on VisDA-C (VisDA). Those methods train with target data and are not source-free, so they are an upper context, not a like-for-like comparison.

Four ablations isolate the key ingredient.

  • Same $K$ heads, no adversaries. Diversity comes only from random seeds.
  • Same $K$ heads, random nuisance groups. Heads are blinded to randomly chosen parameter subsets.
  • One pathway blinded to all groups. This is the "fully invariant" design.
  • Our pathways with Tent's entropy filter in place of agreement.

Every variant gets the same total parameter count, training epochs, and a hyperparameter search of equal size on a held-out synthetic split. No real labels are used for tuning. Beating the field means two things on the real target: higher accuracy than Tent and T3A on the same backbone, and higher accuracy than the seed-diversity ablation. The second comparison is the one that matters most.

Why not just…?

Why not just use a deep ensemble or multi-head model with different seeds and vote? This is the most serious threat. If seed diversity already yields pathways that fail on different images, the adversarial assignment adds nothing. The seed-ablation baseline settles it directly. We also measure how correlated pathway errors are on real images, in both designs. The proposal only stands if the assigned-blindness pathways have clearly less correlated errors on high-confidence real mistakes. If they do not, we will report that and drop the debiasing component.

Why not just make one representation invariant to every simulator factor, via domain randomisation plus full adversarial debiasing? KANFace documents that removing attributes entangled with the task costs accuracy, and that removal is incomplete in wide embeddings (KANFace). Geometry is plausibly entangled with object class in exactly this way. The fully invariant ablation measures this cost on VisDA-C. If it matches our method, the equifinality argument fails on this benchmark.

Why not just run Tent or T3A with a better filter, for example a stricter confidence threshold? A stricter threshold removes uncertain samples, but it keeps confident errors, which are the failure we target. We compare the precision of each selection rule at matched support-set sizes: the fraction of selected real samples that are correct under entropy thresholding versus under cross-pathway agreement.

Why not just translate synthetic images to look real first, as in PixelDA? PixelDA helps with appearance but is limited on structural and geometric gaps (PixelDA). The two approaches are complementary. We run our method on PixelDA-translated sources on digits and LineMod, to check that the gains are not simply appearance gaps that translation would already close.

Proposed Experiments

The main benchmark is VisDA-C: 12 classes, synthetic CAD renders to real COCO and YouTube-BB images, scored by per-class-averaged accuracy. We use re-rendered or proxy nuisance labels as discussed above.

The controlled testbed is digits: SVHN→MNIST and MNIST→MNIST-M. There we generate synthetic variants with known factors, which lets us build a target where we choose which nuisance group breaks. PixelDA reports 98.2% on MNIST→MNIST-M.

For a structural gap, we use synthetic→real LineMod and report mean angle error. PixelDA reports 23.5° against 56.58° for DANN (PixelDA).

GTA5→Cityscapes segmentation is a stretch goal, measured in mIoU against Tent's 35.8%.

We also report three secondary measures on every benchmark. The first is the rate of overconfident errors, meaning errors made with confidence above 0.9. The second is the precision of the support set. The third is behaviour in single-sample streaming, compared with MemBN.

Three decisive checks determine whether the idea holds. First, on the controlled digits target where only texture breaks, the texture-blind pathway's weight should rise and accuracy should hold. The monolithic model should collapse. Second, on VisDA-C, the errors the source-only model makes with high confidence should draw pathway disagreement well above chance. Third, the adversarially assigned pathways must beat the seed-diversity ablation. If the third check fails, the equifinal pathways are just an expensive ensemble, and we say so.