Simulators give perception models free labels: every rendered pixel comes with its object mask, depth and instance identity. To make those images useful, practitioners first translate them to look real, using an image-to-image translator, a network that repaints a synthetic image in the style of real photos. The labels only stay valid if the translator does not move object edges. In practice translators trained with adversarial and cycle-consistency losses do move them. They blur silhouettes, stretch contours and bleed depth across objects, which silently corrupts the labels (AugGAN, SRDA, T²Net). Transformer translators could supply the scene-wide context needed for consistent style. Their cost, however, grows with the square of the number of image patches, so at driving-scene resolution one ends up either shrinking the image, which blurs edges, or windowing attention, which loses global context.
We propose a translator whose attention budget is set by the source labels themselves. Patches in a thin band around synthetic mask boundaries keep full resolution and attend densely. Patches inside a labeled region are pooled into a few region tokens that carry style and global context. This is possible because in the data-augmentation setting every image being translated is synthetic, so its masks are always available at translation time. The claim has two parts. First, label-corrupting errors concentrate on boundaries. Second, at a fixed compute budget, putting resolution on source-label boundaries gives better downstream labels than learned saliency, uniform downsampling or windowing. It also changes how cost grows: the number of full-resolution tokens scales with object perimeter rather than image area.
Core Claim
Let $N$ be the number of patches, $N_b$ the number that fall within $b$ patches of a source mask boundary, $M$ the number of pooled region tokens (one per intersection of a segment with a coarse grid cell), and $w$ the local window side. One translator block then costs roughly
$$ C \approx N_b\,(M + w^2) + M^2 + N\,c_{\text{mlp}}, $$
where $c_{\text{mlp}}$ is the per-patch cost of a lightweight update applied to interior patches. At a fixed scene layout, boundary length grows linearly with image side length, so $N_b \propto \sqrt{N}$. Doubling resolution therefore roughly doubles the expensive term instead of multiplying it by sixteen, as dense attention does. The scientific claim is not the efficiency by itself. It is that this allocation is the right structural prior: it should keep label alignment close to full attention while beating equal-budget alternatives that allocate tokens by content saliency or not at all.
Relation to Prior Work
The mechanism borrows from three efficient-attention designs, each built for a different purpose. SG-Former learns a significance map and keeps fine tokens in salient regions. TCFormer clusters tokens by feature similarity and reports a +6.2 AP gain on hands, which shows that non-uniform token density helps fine structure. LookupViT lets a small set of compressed tokens run full attention and exchange information with many dense tokens through cross-attention; its authors note it was never tested on dense prediction. All three decide allocation from learned image content and were evaluated on recognition tasks. We decide allocation from known geometry, and we evaluate whether labels survive a change of domain.
Pruning work constrains the design. FastV and Which Tokens to Use? drop tokens that attention scores mark as redundant, with little loss in question answering. The transition zones such scores tend to discard are exactly where label misalignment occurs, so we test attention-score pruning as a named competitor rather than assuming it transfers.
The structure-preservation literature motivates the problem and supplies the strongest systems. StawGAN adds a mask-predicting shape controller and a structural-similarity loss to a convolutional generator. Neural Wireframe Renderer forces a shared encoder to reconstruct the input wireframe. DESC regularizes adapted depth with a semantic model and height-based pseudo-labels. These methods add losses or extra branches on top of uniform-grid computation. We change where the model computes, and the approach is compatible with their losses.
The closest bridging works, SatDreamer360 and EpiDiff, also restrict attention using geometry. They restrict it along epipolar lines between camera views, for multi-view synthesis rather than label-preserving adaptation, and they do not reallocate tokens within a single image.
Research Program
The translator is an unpaired, contrastive-style generator with a transformer body. From the source masks we compute a boundary band of width $b$ at patch resolution. Band patches are fine tokens. Every other patch is assigned to the region token of its segment within a coarse cell. In each block, fine tokens attend to their local window and to all region tokens, and region tokens attend to each other. Interior patches receive their region token's output through cross-attention plus a per-patch update, so texture can still vary inside a region. The losses are the adversarial and contrastive patch losses of contrastive unpaired translation for synthetic-to-real, with no extra structural loss in the main model. This keeps any gain attributable to the allocation. A separate variant adds a StawGAN-style mask-consistency loss to test whether the two effects combine.
The analysis has three parts. First, a diagnostic on existing translators: we measure what fraction of label error, defined as the disagreement between a segmenter run on the translated image and the source mask, falls within $b$ patches of a boundary. This tests the premise directly. If error is not boundary-concentrated, the idea fails early. Second, we compare allocation strategies under matched budgets. Third, we sweep resolution to check the $\sqrt{N}$ cost prediction against measured memory and latency.
Baselines
The naive reference trains the downstream depth network directly on untranslated Virtual KITTI and tests on KITTI. No reported result was found on Lacuna for this setting, so we run it ourselves. The standard translation reference is contrastive unpaired translation with a convolutional generator, following Synthetic-to-Real Domain Adaptation using Contrastive Unpaired Translation (no reported Virtual KITTI to KITTI number found on Lacuna). It is included because it is what practitioners actually use.
The strongest translation-based depth method in the file is T²Net, with Abs Rel 0.173 as reported in DESC. The strongest overall system is DESC itself, at Abs Rel 0.156, and 0.145 when combined with S2R-DepthNet. DESC uses semantic and pseudo-label machinery beyond translation. We therefore compare against it in two ways: our translator alone, and our translator substituted into the DESC pipeline in place of its stylization step. Beating T²Net's 0.173 with translation alone is the minimum bar. Improving DESC's 0.156 when our translator is dropped into its pipeline is the meaningful win.
The ablations share one transformer body, identical training data, identical steps, and an equal hyperparameter search of eight configurations each. Every model is held to the same measured FLOPs per image at each resolution. Four ablations isolate the key ingredient by changing only how the fine tokens are chosen: an SG-Former-style learned significance map, TCFormer-style feature clustering, FastV-style attention-score retention, and a random choice of the same number of tokens. Two further baselines show what the budget costs: uniform downsampling to the same token count and Swin-style windowing. Full dense attention with an efficient exact kernel serves as the quality ceiling where memory allows.
Beating the ablations means lower boundary label error and lower downstream Abs Rel than every equal-budget allocation. Reaching full attention's downstream Abs Rel within noise at a fraction of its cost is the efficiency claim.
Why not just…?
Why not just use a convolutional translator with a structural loss, as StawGAN does? This is the real threat. Convolutions are local and cheap, and StawGAN reaches 84.27% Dice on DroneVehicle against 79.34% for TarGAN. The case for attention is that global style consistency matters in large scenes, but the file does not prove that for translation. We therefore include a CUT generator with a mask-consistency loss at matched compute. If it matches our downstream Abs Rel and boundary error, the conclusion is that allocation adds nothing beyond a good loss, and we would report that result.
Why not just downsample uniformly or use windowed attention? Uniform coarsening blurs exactly the contours that carry labels. TCFormer's gains on small structures over uniform grids point that way, and SG-Former's reported gains over Swin (+3.0 mIoU on ADE20K) suggest windowing loses useful context. Neither result comes from translation, so the matched-budget comparison, broken down by boundary error, settles it here.
Why not just let the model learn where to spend tokens, as SG-Former does? Learned saliency optimizes the training loss. In unpaired translation that loss is adversarial and gains nothing from edges staying put, which is the failure mode in the first place. Source labels encode what downstream use requires. The learned-significance ablation tests this directly. If learned allocation matches boundary allocation, the labels were not needed as a prior.
Proposed Experiments
The primary benchmark is Virtual KITTI to KITTI monocular depth. We train a fixed depth network on translated synthetic images and report Abs Rel on KITTI. Alongside it we report boundary F-score between source masks and a frozen segmenter's output on translated images, and FID. The secondary benchmark is the Wireframe dataset, where the wireframe is the structure: we report structural AP against Neural Wireframe Renderer's reported 64.1 sAP₁₀ and 70.73 FID, deriving the boundary band from wireframe lines. The third benchmark is DroneVehicle thermal-to-RGB, using box-derived masks and compared with StawGAN's 84.27% Dice. For efficiency we sweep input side length from 512 to 2048 on driving scenes, plot measured memory and latency against $N$, and fit the growth exponent of the fine-token count.
The decisive checks are three. The diagnostic must show that most label error lies in the boundary band. Boundary-guided allocation must beat random, saliency-based and attention-score allocation at equal fine-token count. The measured cost exponent must sit well below dense attention's.
If the second check fails while the first holds, the premise is right but allocation is the wrong lever, and the structural-loss convolutional baseline becomes the recommended method.
References
- AugGAN: Cross Domain Adaptation with GAN-Based Data Augmentation
- SRDA: Generating Instance Segmentation Annotation via Scanning, Reasoning and Domain Adaptation
- T ^2 2 Net: Synthetic-to-Realistic Translation for Solving Single-Image Depth Estimation Tasks
- SG-Former: Self-guided Transformer with Evolving Token Reallocation
- Not All Tokens Are Equal: Human-centric Visual Analysis via Token Clustering Transformer
- LookupViT: Compressing Visual Information to a Limited Number of Tokens
- An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models
- Which Tokens to Use? Investigating Token Reduction in Vision Transformers
- StawGAN: Structural-Aware Generative Adversarial Networks for Infrared Image Translation
- Neural Wireframe Renderer: Learning Wireframe to Image Translations
- DESC: Domain Adaptation for Depth Estimation via Semantic Consistency
- SatDreamer360: Multiview-Consistent Generation of Ground-Level Scenes from Satellite Imagery
- EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusion
- Synthetic-to-Real Domain Adaptation using Contrastive Unpaired Translation