Models trained on rendered images, such as scenes from a video game engine, often fail on real photographs. One plausible reason is that a Vision Transformer (a network that splits an image into patches and lets every patch "attend to", or read information from, every other patch) learns to label an object by reading the scene around it rather than the object itself. It picks up cues such as the uniform lighting, clean skies and textures, and fixed layouts that a renderer produces. Those context cues change completely in real images. We propose to use a signal that simulators provide for free and real data almost never does: exact per-pixel object boundaries. During training on synthetic data, we run a second forward pass in which patches may attend only to patches of their own object or region. This is the same block-diagonal "don't look across the boundary" masking that language models use to keep packed documents from contaminating each other, applied here to objects inside one image. A consistency loss then asks the ordinary, unmasked network to reach the conclusions that the isolated pass reaches from within-object evidence alone. At test time the deployed model is a standard Transformer with no masks and no added cost.
The two-sentence version is this. Language-model training already knows how to stop tokens from leaning on unrelated neighbouring context, and simulators hand us the boundaries needed to do the same thing inside an image. We use that isolation as a teacher during synthetic training, so the real-world model relies less on scene-context shortcuts that do not survive the move to real data.
Core Claim
The claim has three parts. First, part of the synthetic-to-real gap in Transformer segmenters lives in cross-segment attention: synthetic-trained models route more of an object's evidence through surrounding context than they should. Second, this reliance can be measured by comparing how much attention crosses object boundaries on synthetic versus real images. Third, training with simulator-derived segment isolation as a consistency teacher reduces this reliance and improves zero-shot real-world segmentation. The gains should be largest on small and thin object classes, whose appearance is most easily confused with their surroundings.
The new ingredient is the source and use of the mask. In the packing literature, masks are an efficiency device that keeps separate instances apart. Examples are block-diagonal document masking in Fewer Truncations Improve Language Modeling, which packs whole documents and masks across them, and in How to Train Long-Context Language Models (Effectively), which disables cross-document attention during long-context training. Per-image masks in Patch n' Pack: NaViT and Select and Pack Transformer do the same job for packed images. None of these works uses masks to change what a model learns to depend on across a distribution shift. On the domain-gap side, Adversarially Masking Synthetic to Mimic Real masks input points in LiDAR scans so that synthetic scans resemble sparse real ones, while DREAM projects a synthetic-versus-real subspace out of feature vectors. Neither touches which tokens may exchange information inside the network. Our proposal sits in that unoccupied spot. It differs from the packing work in purpose (a robustness teacher rather than throughput) and from the domain-gap work in mechanism (attention connectivity rather than input corruption or feature projection).
Mechanism
Take a synthetic image $x$ with its simulator label map. We derive segments $s(i)$ for every patch $i$ from connected components of the semantic map, and use instance identities where the dataset release provides them. Driving in the Matrix shows that such identities can be read directly from the game engine's stencil buffer. A patch that straddles a boundary is assigned to its majority segment.
The isolated pass uses the mask $M_{ij}=0$ if $s(i)=s(j)$ and $-\infty$ otherwise. This mask is applied only in a middle band of layers $\ell \in [k_1,k_2]$; early layers keep full attention for local texture and late layers keep it for legitimate context. A kernel detail makes the connection to packing exact. If we sort patch tokens by segment ID and keep their positional embeddings attached, the arbitrary 2D mask becomes a block-diagonal mask over a 1D sequence. That is precisely the packed-document case, so variable-length fused attention kernels built for document masking apply directly, and the isolated pass costs no more than a normal one.
The training objective on synthetic data is
$$\mathcal{L} = \mathrm{CE}(f_{\text{full}}(x), y) + \mathrm{CE}(f_{\text{iso}}(x), y) + \beta \,\frac{1}{|\mathcal{T}|}\sum_{i \in \mathcal{T}} \mathrm{KL}\big(\mathrm{sg}[p_{\text{iso}}(i)] \,\|\, p_{\text{full}}(i)\big),$$
where the terms are defined as follows:
- $f_{\text{full}}$ and $f_{\text{iso}}$ are the same network run with full attention and with the segment mask.
- $y$ is the synthetic label.
- $p_{\text{iso}}(i)$ and $p_{\text{full}}(i)$ are the per-patch class distributions from the two passes.
- $\mathrm{sg}$ stops gradients through the teacher side.
- $\mathcal{T}$ is the set of patches where the isolated pass is correct and confident.
- $\beta$ weights the consistency term.
Restricting the consistency term to $\mathcal{T}$ matters. We do not force context-free predictions everywhere. We only require that, when an object is identifiable from its own pixels, the full model does not override that evidence using the surrounding scene. A variant we also test replaces the second pass with stochastic isolation: with some probability, the main pass itself uses the mask.
Research Program
The work proceeds in three stages. The first is diagnostic, and it answers the open question raised by the tension between VLTSeg and the leakage findings. VLTSeg argues that large vision-language pretraining keeps full fine-tuning on GTA5 from latching onto synthetic quirks. Delimiter Token Scaling, by contrast, finds that pretrained vision-language models leak a large share of attention across image boundaries. The file reports roughly a 50% reduction in cross-image attention once delimiters act as local attention sinks.
To test whether such leakage also appears within an image, we fine-tune ViT backbones on GTA5, then measure cross-segment attention mass per layer and per class on held-out GTA5 images and on Cityscapes. For Cityscapes, the segments come from human labels, which we use only for analysis. The model family starts with a CLIP-initialised backbone matching VLTSeg's recipe, together with a smaller ViT. We also measure a causal quantity: how much a token's prediction changes when its cross-segment attention is ablated at test time. If synthetic-trained models are no more context-dependent on real images than on synthetic ones, the premise is weakened and we report that result.
The second stage is the training method above. We sweep the masked layer band, $\beta$, and the confidence threshold, using GTA5 and SYNTHIA as sources.
The third stage checks whether the effect is specific to true object boundaries. Here we swap simulator segments for random segments of matched size distribution and for coarse superpixels.
Baselines
The naive reference is plain synthetic fine-tuning of the same backbone with full attention, the same augmentations, the same schedule, and the same number of optimizer steps. Because our method runs two forward passes per synthetic image, this baseline is also run at matched FLOPs, meaning twice the steps. That way any gain cannot be attributed to extra compute.
The strongest existing method for the zero-shot setting is VLTSeg. It fully fine-tunes an EVA-CLIP encoder with a Mask2Former decoder on synthetic data and reports a 63.1% mean mIoU across Cityscapes, BDD100k, Mapillary and ACDC when trained on GTA5 alone. We will re-run its recipe with its EVA-02-L backbone and add our loss on top, with identical data, schedule and hyperparameter search budget (the same number of trials for $\beta$ as for its learning rate). Beating it means exceeding its reproduced four-dataset mean by more than the seed-to-seed spread across three seeds. We will state both the reproduced number and the reported 63.1%. We will also run both on a small ViT, where VLTSeg's own concern about deployment cost is sharpest.
As a reference for methods that do use unlabeled real images, Unsupervised Contrastive Domain Adaptation, which uses class prototypes and pseudo-label expansion, reports 60.2% mIoU on GTA5→Cityscapes and 56.5% on SYNTHIA→Cityscapes. That is a different setting from ours, so we do not claim to beat it. Instead we test whether adding our synthetic-side loss to a prototype-alignment adaptation run improves it, which would show the two are complementary.
The key ablation replaces simulator segments with random segments of the same count and size distribution, keeping everything else fixed. If random masks match true masks, then the effect is generic attention regularization and the simulator-boundary claim fails.
Why not just…?
Why not just train on cropped or copy-pasted objects, isolating context at the pixel level? This is the most serious alternative, because a masked-out background is close to a crop. It differs in two ways. The crop removes context from the model entirely, whereas our method keeps full-context pathways and only disciplines them through the confidence-gated consistency term. And copy-paste onto real backgrounds introduces boundary and lighting discontinuities of the kind that DREAM shows networks exploit as a synthetic-versus-real shortcut. In the LiDAR setting, the mixing baseline CoSMix reached 20.0% mIoU against 22.8% for learned masking in Adversarially Masking Synthetic to Mimic Real. That comparison is suggestive, not decisive. We will therefore run two compute-matched variants: a crop-consistency version (the full-image prediction must agree with predictions on object crops) and a class-mix variant. If crop-consistency matches our method, the honest conclusion is that masking is merely a cheaper implementation of cropping. That would still be useful, but it is a smaller claim.
Why not just use random attention dropout? If the gain comes from generic noise in attention, any sparsifying mask would do. The random-segment ablation in the Baselines section is designed to settle this. We also add random per-edge attention dropout at matched sparsity.
Why not just rely on a bigger vision-language backbone? VLTSeg's result is strong, and scale may already remove most context shortcuts. The diagnostic stage tests this directly. If cross-segment dependence on real images is no higher than on synthetic images for EVA-02-L, we expect gains to appear only for small backbones, and we will report the result that way. The 304M-parameter footprint that VLTSeg acknowledges keeps the small-backbone regime practically important either way.
Why not just use soft isolation, such as delimiter-sink scaling? Delimiter Token Scaling reduces leakage at inference without any masks. It needs explicit delimiter tokens, however, and objects inside an image have none. We will test a learned per-segment sink token as a soft variant of our mask.
Proposed Experiments
We train on GTA5 and SYNTHIA and evaluate zero-shot on Cityscapes, BDD100k, Mapillary and ACDC, reporting per-dataset and mean mIoU over three seeds, with both a large and a small ViT. The first decisive check is class specificity. Gains should concentrate on the thin and small classes (pole, traffic light, sign, bicycle) that the contrastive adaptation work singles out as hard, and should not appear on large stuff classes such as road and sky. A uniform gain across all classes would point to generic regularization. The second check is mechanistic. The gap in cross-segment attention mass between synthetic and real inputs, measured in stage one, should shrink under our training and should track the mIoU improvement across the layer-band sweep. The third check is the ranking of true segments against random segments against superpixels, which determines whether simulator boundaries carry the effect. We will report training-time overhead, measured with the sorted block-diagonal kernel. Inference cost is unchanged by construction, since the deployed model uses full attention.
References
- Fewer Truncations Improve Language Modeling
- How to Train Long-Context Language Models (Effectively)
- Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution
- Enhancing Vision Transformers for Object Detection via Context-Aware Token Selection and Packing
- Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation
- Breaking the Synthetic-Real Domain Shortcut for Training-Free Generative Replay-based Class Incremental Learning
- Driving in the Matrix: Can virtual worlds replace human-generated annotations for real world tasks?
- Strong but Simple: A Baseline for Domain Generalized Dense Perception by CLIP-Based Transfer Learning
- Enhancing Multi-Image Understanding through Delimiter Token Scaling
- Unsupervised Contrastive Domain Adaptation for Semantic Segmentation