Teaching a computer to label every pixel of a street scene ("semantic segmentation") requires training images in which every pixel is already labeled. Labeling a single real urban frame by hand takes more than 90 minutes (SHADE). The usual workaround is to train on video-game frames, whose labels come for free from the rendering engine, and then adapt to unlabeled real images (Playing for Data, SYNTHIA). The strongest such pipelines do this at full resolution, because thin objects such as poles, traffic lights, and signs disappear when images are shrunk. They cope with the memory cost by cutting the image into square crops: a small low-resolution crop for context and a square high-resolution crop for detail, each processed separately, with the two predictions blended afterwards (HRDA). We argue that this square-crop habit is itself a source of error. Thin structures are elongated: a pole is a tall sliver and a curb is a long horizontal band. A square detail window cuts both, and the detail pixels never directly "see" the scene context that would tell road from sidewalk.
We propose to replace the two separate square crops with a single sequence of image patches. This sequence mixes coarse patches covering the whole frame with fine patches drawn from rectangular regions of whatever shape fits the structure, such as tall strips, wide bands, or small boxes. Every patch carries its fractional position within the full frame and its scale. A transformer attends across all of them jointly. The token-packing machinery that lets vision transformers handle images of any shape (NaViT) is repurposed to pack several resolutions of the same image, rather than several different images. On unlabeled real images, the fine regions are placed where the coarse pass is uncertain, so pseudo-labels (the model's own predictions, used as training targets) are made in one pass instead of by overlapping sliding windows.
Core Claim
Under a fixed token budget, three properties together should improve synthetic-to-real transfer on thin and elongated classes over two-branch square cropping. The first is joint attention between whole-frame coarse tokens and fine tokens. The second is detail regions whose aspect ratio matches the structure. The third is uncertainty-guided placement on the real domain. We also predict that this design reduces the "texture trap" identified by HRDA: high-resolution synthetic textures are artificial and mislead the model. Our design spends fine tokens only on a minority of the frame, and every fine token is conditioned on shape-level context. The claim is falsifiable piece by piece. If HRDA given elongated crops of equal pixel count matches us, the joint sequence is unnecessary. If square fine regions inside our sequence match us, the aspect ratio of the regions is irrelevant.
Positioning
The motivating evidence comes from two places. HRDA shows that full-resolution adaptation beats half-resolution adaptation by a wide margin, and that isolated detail crops lose context. DAFormer shows that transformer encoders transfer better than CNNs because they rely more on shape than on texture, which suggests shape context should reach the detail pixels directly. HRDA also constrains the design: its finding that high-resolution synthetic texture misleads means we cannot simply run everything at full resolution. Deep High-Resolution Representation Learning (HRNet) keeps a full-resolution stream throughout the network. We instead keep high resolution only where it is spent, which avoids paying HRNet's dense cost.
The method borrows from the aspect-ratio literature. NaViT packs variable-shape patch sequences with attention masks and factorized x/y position embeddings, but it was evaluated only in supervised settings and packs different images rather than different scales of one image. NARAIM supplies fractional position embeddings that encode relative location within the frame, and it shows that aspect-distorting random crops hurt representation learning. LLaVA-UHD slices images into shape-adaptive grids for language-vision models and reports that rigid slice boundaries cause errors on objects that straddle them. That observation is our concern about square detail crops, transferred to dense prediction. The grasp-detection result in Deep Learning for Detecting Robotic Grasps, where aspect-preserving padding beat stretching by over 25%, is further evidence that geometry distortion matters for spatial tasks. None of these works studies domain adaptation.
Geometry-based adaptation is complementary rather than competing. GANDA aligns 3D point features built from predicted depth. D4+DBST injects self-supervised depth. Both run on conventional crops, and our tokenizer could sit underneath either.
Research Program
Encoder. We build a plain, non-hierarchical vision transformer, following the plain-backbone line of Chen et al.. Each token's position embedding has three parts: factorized fractional x and y coordinates in the full frame, a scale embedding for patch stride, and nothing else. The coordinates follow the NARAIM/NaViT approach, so coarse and fine tokens covering the same location agree on where they are. A lightweight decoder upsamples fine tokens within their regions and coarse tokens elsewhere. Where the two overlap, a learned per-pixel gate combines them, analogous to HRDA's scale attention but applied after joint attention rather than in its place.
Detail-region proposals. Each image has a fixed token budget $B$, split into a whole-frame coarse set and a fine set. The fine set is spent on $k$ rectangles whose width and height are free subject to the budget. On synthetic source images, rectangles are sampled around rare and thin classes using the free labels, extending DAFormer's rare-class sampling to region shape. On real target images, a coarse pass produces a per-pixel entropy map, and rectangles are fit greedily to high-entropy connected components, keeping each component's bounding-box aspect ratio. Several images are packed into one batch sequence with NaViT-style attention masks.
Self-training. Pseudo-labels come from the same single sequence, with no overlapping windows. We otherwise keep DAFormer/HRDA's teacher–student averaging, confidence weighting, and ImageNet feature-distance regularizer, so that differences come from tokenization alone.
Analyses. The first analysis is a texture-trap probe: we vary the fraction of the budget given to fine tokens on source images only, and separately low-pass filter source fine tokens, then track target accuracy. The second is pseudo-label quality versus compute: because Cityscapes train labels exist, we can score pseudo-labels for analysis, though never for training. The third is per-class accuracy as a function of object elongation.
Baselines
The naive reference point is a source-only model. It is the same encoder trained on GTA5 with free labels, using aspect-preserving resize and square crops, and is evaluated on Cityscapes with no adaptation. It shows how much the whole adaptation machinery buys.
The low-resolution adaptation reference is DAFormer, which reports 68.3 mIoU on GTA5→Cityscapes. We run it from released settings, and also re-implement it on our plain encoder to separate backbone effects from tokenization effects.
The strongest direct competitor is HRDA, which reports 73.8 mIoU on GTA5→Cityscapes and 65.8 on Synthia→Cityscapes at full 2048×1024 evaluation. It is the strongest competitor because it addresses exactly our problem, keeping detail without losing context, with the same self-training recipe. For fairness, we reproduce HRDA on its original encoder and also re-implement its two-branch scheme on our plain encoder. In both cases its two crops receive the same total token count as our budget $B$, with an identical number of training iterations, identical teacher and loss settings, and an equal hyperparameter-search budget of the same number of runs per method on a held-out split of Cityscapes train. Beating it means a mean gain over three seeds that exceeds seed spread, concentrated on pole, traffic light, traffic sign, rider, and bicycle IoU and on boundary F-score, at equal or lower training and pseudo-labeling cost.
GANDA reports 74.5 mIoU on GTA5→Cityscapes and 67.0 on Synthia→Cityscapes, but it uses predicted depth. We treat it as an orthogonal upper reference: we compare against its number and, if time allows, add our tokenizer beneath its geometry branch.
Three ablations isolate the key ingredients. The first gives HRDA elongated crops with the same pixel count, isolating crop shape from joint attention. The second uses our sequence with fine regions forced to be square, isolating aspect adaptivity. The third uses our sequence with randomly placed fine regions on the target, isolating uncertainty-guided placement.
Why not just…?
Why not just give HRDA rectangular crops? This is the most serious threat, and the first ablation is built to settle it. If HRDA with tall or wide crops of equal pixel count recovers most of our gain on thin classes, the contribution reduces to "crop shape matters." We would report that and drop the joint-sequence claim. We expect it to fall short because HRDA's detail branch still never attends to whole-frame context, which HRDA itself identifies as the cause of road/sidewalk confusion in detail crops.
Why not just downsample, as DAFormer does? HRDA's 73.8 versus DAFormer's 68.3 on the same benchmark already shows that half-resolution discards the detail thin classes need. Downsampling does avoid the texture trap, so our texture-trap probe tests whether a small, context-conditioned fine budget keeps the benefit without the harm.
Why not just run a dense full-resolution network like HRNet with sliding windows? HRNet reaches 81.1 mIoU on Cityscapes with full supervision, but dense full resolution maximizes exposure to synthetic texture and multiplies pseudo-labeling cost. We include a full-resolution sliding-window variant of our encoder, matched in compute, to check whether spending tokens everywhere beats spending them selectively.
Why not just translate synthetic images to look real first? PixelDA handles low-level appearance shifts and was shown only at low resolution. DAFormer-style self-training already outperforms translation pipelines. Translation addresses appearance, not where detail capacity goes, so it is compatible with, rather than an alternative to, our tokenizer.
Proposed Experiments
The main experiments are GTA5→Cityscapes and Synthia→Cityscapes unsupervised adaptation, evaluated at full 2048×1024 on Cityscapes val. We report mIoU over the standard classes, per-class IoU, and boundary F-score on thin classes, against the baselines above at matched token budget.
The first decisive check compares the three ablations. Each ingredient should contribute; if the square-region version equals the full model, the aspect-ratio part of the claim fails.
The second decisive check is a cross-aspect robustness test. We build evaluation variants of Cityscapes val by cropping to 4:3, 1:1, and portrait 9:16, emulating different target cameras, and evaluate each either natively or squashed to the training shape. A model whose positions are fractional frame coordinates should degrade less on native inputs than HRDA does. Squashed inputs should hurt everyone, quantifying aspect distortion in this regime.
The third check concerns labeling cost. Using SHADE's single-source protocol as a reference (42.42 average mIoU across its three real datasets from GTA5 alone), we test whether the tokenizer helps when no target images are seen at all. We also plot how many hand-labeled Cityscapes frames, fine-tuned on top of each adapted model, are needed to reach a fixed accuracy. This converts improvements into annotation hours saved, at 90 minutes per frame.
We also report wall-clock time and memory for pseudo-label generation against HRDA's overlapping sliding windows. No reported pseudo-labeling cost was found on Lacuna for HRDA, so we measure both methods ourselves on identical hardware.
References
- Style-Hallucinated Dual Consistency Learning for Domain Generalized Semantic Segmentation
- Playing for Data: Ground Truth from Computer Games
- The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes
- HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation
- Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution
- DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation
- Deep High-Resolution Representation Learning for Visual Recognition
- NARAIM: Native Aspect Ratio Autoregressive Image Models
- LLaVA-UHD: An LMM Perceiving Any Aspect Ratio and High-Resolution Images
- Deep Learning for Detecting Robotic Grasps
- Geometry-Aware Network for Domain Adaptive Semantic Segmentation
- Plugging Self-Supervised Monocular Depth into Unsupervised Domain Adaptation for Semantic Segmentation
- A Simple Single-Scale Vision Transformer for Object Detection and Instance Segmentation
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks