Segmentation networks trained on synthetic images learn a cheap rule: pixels with the same color and texture belong together, and a change in texture marks a boundary. In a game engine this rule mostly holds. In real photographs it fails in two opposite ways. A patterned or shadowed object gets split into pieces (over-segmentation), and two different surfaces that happen to share a paint color, such as a white wall meeting a white ceiling, get merged (under-segmentation). This proposal attacks the rule at its source, the training data. We re-texture synthetic scenes using the geometry the simulator already knows, so that during training, texture changes and object boundaries no longer line up in any reliable way. The model then has to find boundaries from shape, shading, and layout.
The key design choice is that the randomization is not uniform. Texture is decorrelated from boundaries only where geometry offers another cue: a crease, a change in surface orientation, or a jump in depth. Where two classes lie on nearly the same plane, as road and sidewalk do, their texture contrast is kept, because texture is the only cue that separates them there. This is a direct response to a disagreement in the literature about whether synthetic texture should be suppressed or learned. The method uses rendered depth and surface normals at training time only. The deployed model is an unmodified RGB segmenter.
Core Claim
Our claim is that the two texture failure modes come from one statistical fact about synthetic training data: texture boundaries and object boundaries almost always coincide. Previous texture augmentations leave this correlation largely intact, or they destroy it in the wrong places.
Global and Local Texture Randomization restyles whole images with painting textures, or restyles random spatial patches, reaching 43.7% mIoU on GTA5→Cityscapes. Its random patches ignore scene structure. That creates texture seams in the middle of flat surfaces, which is exactly the "tiling" artifact that ClevrTex identified as a cause of fragmented segmentations.
Adaptive Texture Filtering takes the opposite approach and smooths texture away. Texture Learning Domain Randomization (TLDR) shows why that is risky: road, sidewalk, and terrain have nearly identical flat shapes, and only texture separates them. TLDR therefore regularizes texture features toward an ImageNet-pretrained network, but it does so uniformly over the whole image. Mütze et al. explain both results: shape and color drive boundaries, while texture helps mainly inside regions. They also find that CNNs and transformers share this cue ordering, so a change of backbone will not remove the problem.
Our augmentation encodes that division of labor in the data, using boundary types the simulator can label for free. The new element is to condition the texture edit on which geometric cue is present at each boundary.
Research Program
We first split each synthetic frame into "physical surfaces." A physical surface is a connected region that lies inside one instance mask, contains no depth jump above a threshold, and has surface normals within a fixed angle of each other. Every boundary between two adjacent surfaces is then labeled as one of three types. A geometric boundary is marked by a depth jump or a normal change (wall to ceiling, car to road). A coplanar semantic boundary separates different classes that share one plane (road to sidewalk). An intra-object facet boundary separates two surfaces of the same instance (a car's hood and door).
Each surface is then re-textured with a style drawn from a style bank, using per-region feature-statistic transfer. We follow the AdaIN-style mechanism of the texture-randomization paper, applied under a mask, and reuse its texture-complexity filter for choosing paintings. The sampling rules are what make the method work:
- Across geometric boundaries, the two sides receive the same style with probability $p_{\text{share}}$. This deliberately produces synthetic "white wall meets white ceiling" cases in which only shading and perspective reveal the boundary.
- Across intra-object facets, the two sides receive different styles with probability $p_{\text{split}}$. This teaches the model that a texture change inside an object is not a boundary.
- Across coplanar semantic boundaries, the styles are always distinct, and each side's style is drawn from a class-specific sub-bank. This keeps texture contrast informative where it is the only available cue.
- A single physical surface is never split, which avoids the tiling artifact.
We will study how $p_{\text{share}}$ and $p_{\text{split}}$ trade off against in-domain accuracy. We will also test a cheap extension: a TLDR-style Gram-matrix texture regularizer applied only to pixels inside coplanar regions. The aim is to combine suppression near geometric boundaries with texture learning where texture matters.
The main source domain is SYNTHIA, whose renders we expect to include depth; we will confirm this before relying on it. GTA5 is used with depth recovered from the engine's depth buffer, the extraction route shown in Driving in the Matrix. If per-frame depth cannot be obtained for the released GTA5 segmentation set, we will use monocular depth estimated on the synthetic frames and report that condition separately.
For indoor scenes we use SUNCG, where Task-Assisted Domain Adaptation already exploited rendered normals. That paper adapts normal estimation using semantic segmentation as an auxiliary "anchor" task. We reverse the direction and use geometry to fix semantic segmentation.
Baselines
The naive reference is the same segmentation network (DeepLabV3 with ResNet-50 and ResNet-101) trained on GTA5 or SYNTHIA with standard color jitter and blur. It uses the same schedule as every other arm.
The strongest existing domain-generalization method on this question is TLDR, which reports 46.5 mIoU (ResNet-50) and 47.58 mIoU (ResNet-101) on GTA5→Cityscapes. It is the strongest competitor because it already handles the flat-class texture problem that our coplanar rule targets. Adaptive Texture Filtering reports an average of 44.24% mIoU (ResNet-101) and 40.51% (ResNet-50) across Cityscapes, BDD-100k, and Mapillary Vistas; it is the strongest suppression-style competitor. Global and Local Texture Randomization, at 43.7% on Cityscapes, is the closest method in mechanism.
We will re-run all three with their released settings on our data splits, backbones, iteration count, and a matched hyperparameter search budget. Because re-implementations often fall short of published numbers, we will report both our reproduced figures and the published ones. We count as beating TLDR only a gain in mean mIoU over the three real targets relative to our reproduced TLDR, with a boundary-F-score gain, averaged over three seeds.
Four ablations isolate the key ingredient. The first uses random rectangular masks with the same style bank. The second uses per-class masks, where every semantic region gets its own random style. The third uses our geometric partition with independent sampling, so there is no deliberate sharing or splitting. The fourth removes the coplanar rule.
For indoor scenes, no reported result for SUNCG→NYUDv2 semantic segmentation was found on Lacuna. Our RGB baseline defines the reference there. The in-domain RGB-D figure of 56.9% mIoU for CMX on NYUDv2, from the RGB-D survey, serves only as an upper reference.
Why not just…?
Why not just randomize texture per semantic region using the simulator's class masks? This is the most serious threat. It is simpler, and it removes the tiling seams. However, it keeps, and even strengthens, the rule that "a texture change means a class change." It never shows the model same-texture adjacent surfaces and never shows it multi-texture single objects. If per-class masks match our full method on Cityscapes mIoU and on the wall/ceiling and fragmentation diagnostics described below, geometry adds nothing, and we will say so. The ablation is designed to settle this question directly.
Why not just train an auxiliary depth or normal head on synthetic data? Multi-task geometry supervision is cheap. CrossMatch, which reports 57.7% mIoU on GTA5→Cityscapes, shows that depth-guided consistency helps, but only in a source-free adaptation setting with target images. An auxiliary head alone leaves the texture shortcut in the data available for the segmenter to use. We will run an auxiliary-normal-head arm, both alone and combined with our augmentation, to test whether the two effects are complementary or redundant.
Why not just translate synthetic images to look real, or scale up the data? Pixel-level translation such as PixelDA matches low-level style, but it keeps the synthetic correlation between texture and boundaries. The texture-randomization paper reports that painting-based randomization outperformed direct style-alignment methods. Scaling data did not fix texture failures in ClevrTex, where 50,000 textured training images still left SPAIR at 0% mIoU. Papa et al. similarly found that extra model capacity improved reconstruction but not segmentation.
Proposed Experiments
The main evaluation is domain generalization from GTA5 and from SYNTHIA to Cityscapes, BDD100K, and Mapillary Vistas. We report per-class IoU and a boundary F-score in trimap bands around ground-truth edges. Two diagnostics decide whether the mechanism works as claimed.
The first is a per-class breakdown. Our method should improve classes whose boundaries are mostly geometric (poles, buildings against sky, vehicles) while leaving road, sidewalk, and terrain no worse than under TLDR. A drop on those flat classes would falsify the coplanar rule.
The second is a fragmentation count: the number of connected predicted components per ground-truth instance, measured on real vehicles and people. This measures over-segmentation directly.
For under-segmentation, we train on SUNCG and test on NYUDv2 and ScanNet++, reporting ceiling and wall IoU and the wall→ceiling confusion rate. OGGSplat reported 0.1% ceiling IoU for its 2D encoder on ScanNet++, which marks this as a known failure case.
As a controlled test, we train on ClevrTex scenes re-rendered with depth and evaluate supervised segmentation on the CAMO split, where objects and background share materials, and on the out-of-distribution split. On CAMO, color contrast at boundaries is zero, so only geometry-derived cues can succeed.
The decisive result would be improvement on the real CAMO-like cases, namely ceiling IoU and fragmentation, together with no loss on the flat classes and a gain over per-class-mask randomization. That combination would show that conditioning texture edits on geometry, rather than just adding more texture variety, is what closes the gap.
References
- Global and Local Texture Randomization for Synthetic-to-Real Semantic Segmentation
- ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentation
- Adaptive Texture Filtering for Single-Domain Generalized Segmentation
- Texture Learning Domain Randomization for Domain Generalized Segmentation
- On the Influence of Shape, Texture and Color for Learning Semantic Segmentation
- Driving in the Matrix: Can virtual worlds replace human-generated annotations for real world tasks?
- Task-Assisted Domain Adaptation with Anchor Tasks
- A Review of Semantic Segmentation Methods Based on RGB-D Images
- CrossMatch: Source-Free Domain Adaptive Semantic Segmentation via Cross-Modal Consistency Training
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks
- Inductive Biases for Object-Centric Representations in the Presence of Complex Textures
- OGGSplat: Open-Vocabulary Gaussian Growing for Expanded Field-of-View