Vision transformers (image models that cut a picture into small patches, or "tokens," and let every patch exchange information with every other patch) transfer from video-game training data to real street scenes much better than convolutional networks. They also degrade less under rain, fog and snow. The catch is cost: computation grows quadratically with the number of patches, and driving-scene segmentation uses many patches. The standard way to cut that cost is to let a small learned module decide, per image, which patches deserve full computation and which can be skipped. This note argues that this decision module is itself a learned model exposed to the domain gap. If it is trained on synthetic images, it will learn what "important" looks like in a game engine. In real, messy images it will then skip thin poles and rare vehicles, and it will spend its budget on bright rain streaks and lens glare.
The proposal is to measure this failure directly and then fix it. The fix has two parts. First, the patch-selection module is adapted to the real domain inside the same teacher–student self-training loop used for unsupervised domain adaptation, meaning adaptation with no real-image labels. Second, skipped patches are "idled" rather than deleted, so the dense pixel grid a segmentation decoder needs stays intact and early mistakes can be undone. In two sentences: when you make a sim-to-real transformer cheaper by skipping patches, the skipping rule should be adapted to the real domain too, because a rule learned on synthetic data is the component most likely to break. We will show how much of the accuracy lost by efficient models under sim-to-real transfer comes from bad skipping decisions, and recover it at a matched on-device latency.
Core Claim
The claim is that most of the extra accuracy loss that token-reduced transformers suffer on synthetic-to-real transfer comes from selection errors, not from lost capacity. It follows that adapting only the selection policy to the target domain should recover most of that loss at no extra inference cost.
A concrete mechanism predicts this, and it has two parts. The first is shortcut behaviour. Synthetic renderings simplify texture, lighting and occlusion (Learning to Estimate Scenes from Images; Learning an object class representation on a continuous viewsphere), so a scorer trained on them can rank patches using cues that do not exist in real images. The second is concentration. In vision-language models, On the Adversarial Robustness of Large Vision-Language Models under Visual Token Compression shows that token compression acts as a "distortion concentrator": it preferentially keeps perturbed, high-magnitude tokens. Are Vision Transformers Robust to Patch Perturbations? shows that a single corrupted patch can draw attention onto itself, giving a 61.5% fooling rate on DeiT-small against 30.6% on ResNet50. Both results were obtained with adversarial perturbations. Our question is whether ordinary real-world corruptions such as weather, glare and sensor noise trigger the same selection bias in a dense, domain-adapted model. If they do, token reduction quietly gives up the robustness that made transformers attractive for sim-to-real in the first place.
What Prior Work Establishes and Leaves Open
Two papers motivate the regime. DAFormer pairs a Mix Transformer encoder with a context-fusion decoder and self-training stabilizers such as rare-class sampling, and reaches 68.3 mIoU on GTA5→Cityscapes. It credits the transformer's shape bias over texture, but it uses the heavy MiT-B5 backbone with a full token grid. Cityscape-Adverse edits Cityscapes with a diffusion model to add rain, fog, snow and night. Under these edits, transformers lose 12.8 mIoU on average while CNNs lose 21.4. This is the robustness margin we must not squander. SegFormer shows that its small variant runs at 48 FPS on Cityscapes and tolerates corruptions, but it computes every token at every stage.
The efficiency line supplies our method and its constraints. IdleViT lets unselected tokens bypass attention and the feed-forward layers and rejoin through the residual path. It cuts compute by 33% on DeiT-S with under 0.2% accuracy loss. We reuse this "idle, don't delete" mechanism because it keeps the 2D grid that segmentation needs. IdleViT was tested only on clean ImageNet classification.
Patch Slimming prunes top-down from a class token, a design that does not apply to dense prediction. Token Fusion shows that naive averaging of merged tokens distorts feature norms in early layers: ToMe reaches 67.54% and norm-preserving fusion 74.06% on ViT-B at r=20. That result constrains us against merging in early layers.
DynaSegFormer makes SegFormer cheaper by gating channels rather than tokens. It saves more than 60% of compute on ADE20K for a 0.5 mIoU drop, and its dynamic gating beats static gating by nearly 3 mIoU. That paper is evidence that instance-dependent decisions matter. It never faces domain shift.
Semantic-aware Message Broadcasting shows that domain shift is regional and uses group tokens to align specific regions. This supports adapting a per-region decision rather than a global one. However, it keeps the full sequence and evaluates on classification.
None of these papers adapts the token-selection policy itself to a new domain. None measures where selected tokens land under real corruption.
Research Program
The system is a MiT-B0/B1 SegFormer-style encoder trained with the DAFormer self-training recipe on GTA5→Cityscapes, plus a lightweight token selector in the high-resolution first two stages, where the token count is largest. In each block, the selector keeps a fixed number $K$ of query tokens, and the rest are idled. Idled tokens carry their previous value forward, so the depthwise convolutions in the Mix-FFN still see a full grid. Using a fixed $K$ rather than a threshold keeps tensor shapes static; this is a deliberate response to the concern that irregular sequence lengths run poorly on edge accelerators.
The novel ingredient is what trains the selector on real images. In self-training, a slowly updated teacher labels real images for the student. We train the selector on target images to rank patches by a teacher-derived importance signal. This signal combines teacher prediction entropy, proximity to teacher-predicted class boundaries, and a boost for rare classes, mirroring DAFormer's rare-class sampling. In effect, the selector learns to spend compute where the real-domain labeler is uncertain or where structure is fine.
To counter concentration, the selector sees normalized features: per-token layer normalization before scoring, with no raw-norm input. On source images, it is trained against a ranking target that deliberately ignores feature magnitude. The hypothesis is that removing magnitude as a usable cue blocks the "bright artifact wins" failure. An ablation checks that hypothesis directly.
The analysis program comes before the method and is valuable even if the method fails. We decompose the efficient model's target-domain loss by swapping selection policies while freezing the backbone. The policies are: a source-trained selector; a target-adapted selector; and an oracle selector that uses Cityscapes ground truth to keep boundary and rare-class tokens. The gap between the source-trained and oracle selectors, at a fixed backbone and a fixed $K$, measures the loss caused purely by selection.
We also measure concentration. We paste synthetic lens-dirt and glare patches at known locations onto Cityscapes and Cityscape-Adverse images, then report the fraction of the keep budget each selector spends on those regions. We compare that fraction with the patches' area share of the image.
Baselines
The naive reference point is the unmodified dense MiT-B0 trained with the full DAFormer recipe on GTA5→Cityscapes. It is the accuracy ceiling for our backbone and the latency ceiling we are trying to beat. There is no reported result found on Lacuna for DAFormer with a MiT-B0 or MiT-B1 encoder, so we will run it ourselves. The only reported anchor is 68.3 mIoU for DAFormer with MiT-B5 (DAFormer), which we include as an upper reference, not as a matched competitor.
The strongest efficiency competitor at matched latency is the dense B0 with a reduced input resolution, chosen so that on-device latency equals our model's. Uniform downsampling needs no selector and has perfectly regular compute, which makes it the hardest honest baseline.
The strongest existing token-reduction method is IdleViT-style idling with its selector trained on source data only. This is the obvious combination of existing parts, and our proposal must beat it. For segmentation there is no reported result found on Lacuna for IdleViT; its only reported figure is the 33% compute reduction on DeiT-S from IdleViT.
We also run a DynaSegFormer-style channel-gating model trained under the same recipe, using DynaSegFormer. Its reported figures are ADE20K-only: 36.9 mIoU at 3.3 GFLOPs versus 37.4 at 8.4 GFLOPs.
The key ablations isolate each ingredient in turn. The first uses the target-adapted selector with the norm-blind input removed. The second uses a source-only selector with a random-keep control. The third uses the oracle selector described above.
All models get the same data, the same schedule of 16 hours on one RTX 2080 Ti per model (DAFormer's reported budget), and the same number of hyperparameter trials. All are measured on the same embedded GPU at batch size one with identical INT8 export.
Beating the baselines means the following. At equal measured latency, we must exceed both the downsampled dense model and the source-trained idling model on Cityscapes mIoU and on rare-class IoU. We must also retain a larger share of the dense model's accuracy on ACDC and Cityscape-Adverse, and those gains must hold across three seeds.
Why not just…?
Why not just use a smaller backbone with the standard adaptation recipe? This is the baseline we take most seriously. DAFormer's per-class gains (Train rising from 16.0 to 65.1 IoU) suggest that rare classes need context capacity, but that evidence compares architectures, not a small model against a larger model run cheaply. Our design keeps MiT-B1 capacity while idling tokens to reach MiT-B0 latency. If dense B0 matches us at equal latency, then the capacity argument fails for this setting, and we will report it as the main result.
Why not just downsample the input uniformly? Uniform downsampling is regular, simple, and loses no information to selection errors. It is exactly what a domain-fragile selector must beat. Patch Slimming reports that learned selection beats uniform pruning on DeiT-S (79.4% versus 77.2%), but only in-domain on classification. Downsampling hurts thin structures most, so the latency-matched comparison, broken down by class for poles, signs and riders, settles it.
Why not just train the selector on source data and rely on the backbone's adaptation? This is our central hypothesis test, not a strawman. The policy-swap decomposition measures directly how much the source-trained selector costs relative to the oracle. If that gap is small, the problem we posit does not exist, and the contribution reduces to the diagnostic.
Why not just quantize or prune channels and keep every token? These are complementary rather than competing. Every model we test is INT8-exported. Channel gating leaves the number of attention queries unchanged in the high-resolution stages, and DynaSegFormer's gate predictors add about 12% parameters. We will report the combination of channel gating and token idling to show whether the savings add up.
Proposed Experiments
The decisive checks are three, all on GTA5→Cityscapes with the Cityscapes validation set, plus zero-shot evaluation on ACDC and Cityscape-Adverse.
First, the policy-swap decomposition at keep ratios of 50% and 30% in stages one and two. If the source-to-oracle selection gap is below 1 mIoU, we stop claiming that selection is the bottleneck.
Second, the concentration measurement. For each selector, we report the ratio of keep budget spent on pasted glare and dirt regions to those regions' area share. We also report robustness retention, defined as adverse-condition mIoU divided by clean mIoU, compared against the dense model. Here the concern is whether reduction erodes the transformer advantage that Cityscape-Adverse documents (12.8 versus 21.4 mIoU drop).
Third, the matched-latency accuracy and latency frontier against dense B0, downsampled B1, source-selector idling, and channel gating. We also repeat the analysis on Synthia→Cityscapes and on VisDA-S to check generality. The strongest evidence for the idea would be a large source-to-oracle gap, most of it closed by target adaptation alone, with the norm-blind scorer measurably cutting budget spent on artifacts.
References
- Learning to Estimate Scenes from Images
- Learning an object class representation on a continuous viewsphere
- On the Adversarial Robustness of Large Vision-Language Models under Visual Token Compression
- Are Vision Transformers Robust to Patch Perturbations?
- DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation
- Cityscape-Adverse: Benchmarking Robustness of Semantic Segmentation With Realistic Scene Modifications via Diffusion-Based Image Editing
- SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
- No Token Left Behind: Efficient Vision Transformer via Dynamic Token Idling
- Patch Slimming for Efficient Vision Transformers
- Token Fusion: Bridging the Gap between Token Pruning and Token Merging
- Dynamically Pruning Segformer for Efficient Semantic Segmentation
- Semantic-Aware Message Broadcasting for Efficient Unsupervised Domain Adaptation