Randomize Only the Textures the Simulator Gets Wrong

Image classifiers and segmenters trained on rendered or game-engine images often fail on real photographs. A common explanation is texture bias: convolutional networks decide mostly from local surface patterns such as fur, bark, or asphalt grain, rather than from object outlines. Simulators render those surfaces with limited texture libraries and simplified shading, so a network that learns synthetic textures has learned the wrong thing. The standard remedy is to repaint every synthetic pixel with random artistic styles, so that texture stops predicting the label. This proposal argues that the remedy is too blunt. Some synthetic textures, such as road asphalt in a driving game, are close enough to reality to be useful cues. Others, such as game-engine foliage or uniform CAD plastic, are not. Erasing all of them throws away real information. The literature already shows that maximally texture-free training can hurt real-world accuracy.

The core claim is that texture randomization should be applied selectively, per semantic class. Its strength should be set by an estimate of how unrealistic that class's synthetic texture is. That estimate can be computed without any real target images: we measure how far the class's synthetic texture statistics sit from the texture statistics of a generic real photo corpus (ImageNet), using a frozen ImageNet-pretrained network. The resulting method is a small change to existing pipelines. Classes whose rendered textures look unlike real ones get heavily restyled. Classes whose textures look realistic are mostly left alone. The method needs one precomputed number per class and runs in the zero-target-access (domain generalization) setting.

What the evidence says and where the gap is

Three findings motivate the idea. ImageNet-trained CNNs are biased towards texture showed that ResNet-50 relies on shape only about 22% of the time on cue-conflict images, and that training on style-transferred images raises this to 81% and improves robustness. Harnessing Synthetic Datasets found what it calls a shape bias paradox: the synthetic pretraining set with the highest shape bias (StyleGAN-Random) gave about 18% downstream accuracy, versus about 35% for the lower-shape-bias Shaders-21k. So stripping texture is not monotonically good. Shape-biased CNNs are Not Always Superior in Out-of-Distribution Robustness showed that the right balance between shape and texture depends on the category: texture matters for animals, shape for rigid artifacts. Its adaptive recombination (STAR) raised average OOD robustness to 57.2% from 50.7% for Stylized-ImageNet. STAR works on natural images, however, and says nothing about simulators.

The closest sim-to-real work all randomizes texture uniformly. Learning Texture Invariant Representation restyles all of GTA5 with paintings and then self-trains on target images. Its stylized-only model (42.5 mIoU) barely beats a CycleGAN-translated-only model (41.0). That small margin suggests realistic texture and texture diversity each carry information. Texture Underfitting for Domain Adaptation mixes AdaIN-stylized and original images 50/50. AdaIN (adaptive instance normalization) is a fast style transfer that swaps feature means and variances. Global and Local Texture Randomization is the nearest competitor. It restyles whole images (GTR), restyles random rectangular masks (LTR), and enforces consistency between the two. Its local masks are random rather than semantic, and its strength is the same for every class. Our proposal differs in exactly those two respects.

Two further works constrain the design. The Fourier methods, FDA and AGFA, perturb global amplitude spectra. FDA needs target images, and neither is class-aware. Towards Sim-to-Real Industrial Parts Classification reports what it calls the albedo trap: parts that share a surface color get confused, and accuracy collapses from 83.2% on isolated parts to about 40–50% on assemblies. That paper only tried background randomization, never randomization of object surfaces.

Core Claim

Per-class sim-to-real error is partly predictable from how unrealistic that class's synthetic texture is. Randomizing texture in proportion to that unrealism beats uniform randomization at the same average strength. It also preserves accuracy on classes whose synthetic textures are faithful.

For each synthetic class $c$, let $d_c$ be the mean k-nearest-neighbor distance from Gram-matrix texture descriptors of class-$c$ pixel regions to a bank of descriptors from ImageNet patches. Gram matrices summarize which feature channels co-activate, and they are the standard texture statistic in style transfer. Both are computed with a frozen ImageNet-pretrained network's early layers. During training, each class-$c$ region in each image is restyled with probability

$$p_c = p_{\min} + (p_{\max}-p_{\min})\,\mathrm{rank}(d_c)/C,$$

where $C$ is the number of classes and $p_{\min}, p_{\max}$ bound the strength. Using ranks rather than raw distances makes the schedule insensitive to descriptor scale. Restyling uses AdaIN with the painting library of the prior work, applied inside the rendered semantic mask. Blending is feathered at the mask edge to limit the boundary damage that Texture Underfitting was criticized for. Semantic masks come free with synthetic data, so no real labels or images are needed. ImageNet is already the pretraining source, which matches the access assumptions of Automated Synthetic-to-Real Generalization.

Research Program

Diagnostic stage. Before training anything new, we test the premise. For a source-only model on GTA5→Cityscapes, SYNTHIA→Cityscapes, and VisDA-17, we compute the per-class gap between synthetic validation accuracy (or IoU) and real accuracy. We then test whether this gap correlates with $d_c$ using Spearman correlation across classes.

We also compute an oracle $d_c^{*}$ using Cityscapes or COCO texture banks. This tells us how much is lost by using ImageNet as a stand-in for "real". We further measure per-class texture reliance by swapping in-mask textures on synthetic validation images and recording the accuracy drop. This separates classes that are unrealistic but not relied upon, where randomizing them should matter little.

Intervention stage. We train with the $p_c$ schedule on the same backbones as prior work: ResNet-50/101 segmenters, a ResNet-101 classifier for VisDA-17, and ConvNeXt for the industrial parts dataset SIP-17. The mask-based restyling operator is swapped between AdaIN and a masked Fourier amplitude mix. This checks whether the benefit comes from class selection rather than from the choice of restyling operator.

Evaluation discipline. All model selection follows BiasBed, which showed that best-epoch selection inflates debiasing gains and that most methods do not beat standard training under Friedman and Nemenyi tests. We therefore select on synthetic validation data only, run five seeds, report last-n-epoch averages, and apply the same tests.

Baselines

Source-only training. This is the do-nothing reference: standard empirical risk minimization with the usual crop, flip, and color jitter augmentations. The reported reference is 22.17% mIoU for FCN-ResNet-50 on GTA5→Cityscapes in Automated Synthetic-to-Real Generalization. The same paper reports 28.2% on VisDA-17, and VisDA reports 21.4 mIoU for segmentation. We rerun it under our protocol.

Uniform texture randomization. This is the strongest competitor: GTR+LTR+CGL from Global and Local Texture Randomization, which reports 43.7% mIoU on GTA5→Cityscapes with zero target access. It answers the same question, zero-target-access texture debiasing, with the same painting library. We reimplement it with our backbone, schedule, painting set, and tuning budget, which is a fixed number of hyperparameter trials selected on synthetic validation for every method. Beating it means a statistically significant mean-mIoU gain across seeds under this protocol. It must also come without a significant loss on any class group, stuff or things.

ASG. Knowledge-preserving generalization via Automated Synthetic-to-Real Generalization is the strongest reported classification method in the file, at 61.1% on VisDA-17. It is run both alone and stacked with our schedule, because the two are complementary.

For SIP-17. The reference is ConvNeXt trained on background-randomized renders, reported at 83.2% on isolated parts and about 40–50% on assemblies in Towards Sim-to-Real Industrial Parts Classification.

Ablations isolating the key ingredient. The first is uniform in-mask randomization at the same mean $p$. It isolates class selectivity from semantic masking. The second is a shuffled schedule that uses the same set of $p_c$ values assigned to random classes, and the third an inverted schedule. These isolate whether the fidelity signal itself matters or whether any non-uniform partial randomization helps. The oracle $d_c^{*}$ gives an upper bound. If the shuffled schedule matches ours, the core claim is false.

Why not just…?

Why not just randomize everything, as GTR/LTR does? This is the real threat. On driving benchmarks, nearly all game textures may be unrealistic enough that $p_c$ is uniformly high, and then selection adds nothing. Two pieces of evidence argue against assuming this: the shape bias paradox and the narrow 42.5 vs 41.0 gap between stylized-only and realism-only training. Neither is decisive for GTA5. The matched-strength uniform ablation settles it. We also report per-class results, expecting gains concentrated in high-fidelity classes such as road and sky that uniform restyling degrades. If no such concentration appears, we will conclude that uniform randomization suffices on these benchmarks.

Why not just use Fourier amplitude randomization (APR or AGFA) instead of style transfer? Amplitude-Phase Recombination cut CIFAR-100-C error from 55.6% to 33.9%, and AGFA reached 68.0% on DomainBed. Both perturb whole images uniformly, however, so they inherit the same blindness to class. We treat the amplitude mix as an alternative restyling operator inside our schedule. If global APR matches class-selective APR, selection is not the active ingredient.

Why not just anchor to an ImageNet teacher, as ASG does? ASG preserves real-world features, but it does not tell the network which synthetic cues to distrust. Its best number, 61.1% on VisDA-17, still leaves a large gap. The stacked run shows whether the two effects add or overlap.

Why not just use a ViT or ConvNeXt? Towards Sim-to-Real Industrial Parts Classification reports that both architectures fall into the albedo trap on assemblies. Err on the Side of Texture finds strong texture association in modern backbones on real data. We run ConvNeXt with and without our schedule to measure whether architecture alone closes the gap.

Decisive Checks

The proposal lives or dies on three results. First, the diagnostic must show a positive per-class correlation between $d_c$ and the sim-to-real gap on at least two of the three benchmarks. If it does not, we stop and publish the negative result. Second, our schedule must beat both the matched-strength uniform and the shuffled schedules under BiasBed-style significance testing on GTA5→Cityscapes and VisDA-17. Third, on SIP-17 assemblies, randomizing object surfaces must lift accuracy above the roughly 40–50% background-only reference. With only 566 real test images, we report bootstrap confidence intervals there. The oracle-versus-proxy comparison then tells us whether ImageNet is an adequate stand-in for "real", which determines whether the method is truly target-free in practice.