Choosing Token Resolution to Shrink the Sim-to-Real Gap in Frozen Vision Transformers

Large self-supervised vision models such as DINOv2 are increasingly used as frozen feature extractors. The model's weights stay fixed, and only a small prediction head is trained on top. This makes them attractive for learning from synthetic data, because a game engine can supply unlimited pixel-perfect labels while the frozen backbone keeps the general visual knowledge it learned from real photographs. There is a hidden knob in this setup, though. A vision transformer (ViT) chops the image into square patches and turns each patch into a "token". Each token therefore summarizes a certain number of pixels. If you resize the input or change the patch size, you change how much fine texture enters each token. Prior sim-to-real work has found that synthetic and real images differ most in their finest textures and their global lighting, and least in mid-scale structure. That suggests the pixels-per-token setting is not just a speed/accuracy trade-off. It may directly control how "synthetic-looking" the frozen features are.

We propose to measure this effect and use it. A segmentation head is trained on frozen-backbone features of synthetic street scenes, extracted at a range of token resolutions. At deployment, the resolution is chosen using only unlabeled real images: we pick the setting at which synthetic and real features are hardest to tell apart while synthetic accuracy is still near its best. The central bet is falsifiable. The token resolution that is best on synthetic validation data is not the one that transfers best to real data, and a label-free domain-discrepancy signal can find the better one.

Core Claim

For a frozen ViT feature extractor with a head trained on synthetic data, real-domain accuracy depends on the pixels-per-token scale. It peaks at a coarser scale than in-domain accuracy does, because fine tokens carry rendering-specific high-frequency texture. A head that is allowed to pick or weight scales using synthetic labels alone will drift toward the fine, synthetic-optimal end. Selecting the scale with an unlabeled-target discrepancy criterion should recover most of the gap between that choice and an oracle that picks the scale using real labels.

The claim rests on three lines of prior work. MRFP shows, for convolutional backbones, that domain variance concentrates at the lowest and highest feature frequencies while mid-level features are domain-invariant. It responds by perturbing features during training. We instead ask whether a frozen ViT's sampling rate already filters the variant band. A Comprehensive Study of Vision Transformers on Dense Prediction Tasks reports a "resolution paradox": ViTs degrade quickly when inference resolution departs from training resolution, because the learned position grid is stretched by interpolation. This is a constraint on our design. It tells us the scale sweep must separate genuine domain effects from interpolation damage. FlexiViT supplies a second way to change pixels-per-token without resizing the image. Its pseudoinverse resizing ("PI-resize") rescales the patch-embedding weights so that a model can run at a different patch size. FlexiViT trains for this ability during pre-training. We apply PI-resize post hoc to a frozen model, which is untested, so it enters as an ablation rather than the core.

Research Program

The backbone is a frozen DINOv2 ViT (DINOv2), whose patch features are known to be linearly separable for segmentation without fine-tuning. On top of it sits a light head, either a linear layer or a two-layer decoder, trained on GTA5 frames with segmentation labels. Pixels-per-token is varied along two routes. The first resizes the input image while keeping the native patch size and interpolating position embeddings. The second keeps the image fixed and PI-resizes the patch embedding to larger or smaller patches. A third route keeps the token count fixed by tiling crops, which avoids position interpolation entirely. It serves as the control that separates domain effects from the resolution paradox.

For each scale $s$ we record three quantities. The first is synthetic validation mIoU, $A_{syn}(s)$. The second is real-domain mIoU on Cityscapes, $A_{real}(s)$, which is used only for analysis and for the oracle. The third is a domain separability score $D(s)$: the held-out accuracy of a linear probe trained to classify patch tokens as synthetic or real, using unlabeled Cityscapes images. A probe accuracy is comparable across scales in a way that raw feature distances are not, since raw distances shrink trivially when there are fewer tokens. The selection rule is

$$s^* = \arg\min_{s} D(s) \quad \text{subject to} \quad A_{syn}(s) \ge \max_{s'} A_{syn}(s') - \delta,$$

where $\delta$ is a fixed tolerance in mIoU points chosen once on Synthia→Cityscapes and then frozen for all other pairs.

The head is trained on features drawn at randomly sampled scales, so a single head works at every candidate $s$. This also tests whether scale mixing acts as a diversity axis that costs no new rendering.

Two diagnostics explain why the curve looks the way it does. The first is a frequency analysis: we low-pass and high-pass filter synthetic inputs and check whether the fine-scale gap disappears when high frequencies are removed, as MRFP's account predicts. The second concerns background "register" tokens. These are high-norm tokens that DINOv2 places in low-information regions such as skies, documented in Randomized-MLP Regularization Improves Domain Adaptation and Interpretability in DINOv2. We count them per scale and per domain, because rendered skyboxes may attract them differently from real skies. If register behavior rather than texture drives the scale dependence, the story changes, and we will report that.

The diversity question is addressed by repeating the sweep with 10%, 25% and 100% of GTA5. Driving in the Matrix found a threshold of diversity: 200k diverse synthetic frames beat a Cityscapes-trained detector on KITTI. Generalizing from Synthetic Data to the Real World argues that more data from the same simulator mostly buys more shortcuts. If the gap at fine scales comes from rendering texture, the advantage of a coarser $s^*$ should not shrink as synthetic data grows. If it does shrink, then scale selection is merely compensating for too little data.

Baselines

The naive reference point is the frozen DINOv2 with its head trained and tested at the default resolution with sliding-window inference on GTA5→Cityscapes. We found no reported result on Lacuna for a frozen DINOv2 head trained on GTA5 and tested on Cityscapes, so we will run it ourselves. Our selected scale must beat this by a margin larger than the seed-to-seed standard deviation over three seeds.

The strongest existing sim-to-real methods in the file serve as external reference points. On a different backbone, Global and Local Texture Randomization reports 43.7% mIoU on GTA5→Cityscapes. MRFP reports 39.27% average mIoU across its target domains with a ResNet-50 trained on GTAV/Synthia, against a 31.21% vanilla baseline. The VisDA segmentation challenge winner reached 47.5 mIoU, starting from a 21.4 source-only baseline. These numbers come from fully trained convolutional networks, so they place our results in context but do not give a controlled comparison.

The controlled strong competitors are run by us on the same data, the same backbone family, and the same head-tuning budget. The first is full end-to-end fine-tuning of DINOv2 on GTA5. The second is the frozen head trained on GTR/LTR-stylized GTA5. The third is standard multi-scale test-time averaging over all scales. The fourth is a head that learns its own scale weighting from synthetic labels. Selection beats these if it matches or exceeds them on real-domain mIoU without access to real labels. The upper bound is the oracle scale chosen with Cityscapes labels.

The ablation that isolates our key ingredient replaces $D(s)$ with pure synthetic-validation selection. If the two rules pick the same scale, the claim fails. A second ablation swaps input resizing for PI-resize and for fixed-token tiling. It tests whether the effect comes from pixels-per-token itself or from how that rate is reached.

Why not just…?

Why not just average predictions over many scales, as standard test-time augmentation does? This is the real threat, because averaging can dilute the bad scales without choosing any. Averaging does give equal vote to fine scales, which we predict carry the most synthetic-specific texture. We will report both methods head to head. If averaging matches selection, the honest conclusion is that scale diversity at test time matters and choosing a scale does not, and we would report it that way.

Why not just fine-tune the backbone on synthetic data? Fine-tuning lets the backbone absorb simulator shortcuts, which is the failure mode Generalizing from Synthetic Data to the Real World describes. Frozen extraction has been both competitive and much cheaper for domain adaptation: PRPL reports 81.2% on Office-Home while running about 18× faster than CAN. That evidence comes from classification, however, so our end-to-end fine-tuning run is what settles the question for dense sim-to-real.

Why not just stylize the synthetic images instead? Stylization attacks texture at the input. If texture is the cause of the scale effect, a stylized source should flatten the $A_{real}(s)$ curve. That would be informative, not fatal. We test stylization alone and combined with selection, and report whether the two add up or overlap.

Why not just tune the scale on a few labeled real images? That is the oracle, and it is a legitimate method whenever real labels exist. Our contribution is limited to the label-free case, and we report how much of the oracle gain selection recovers.

Proposed Experiments

The primary transfer is GTA5→Cityscapes, with Synthia→Cityscapes as the pair used to set $\delta$, and GTA5→Nexar from VisDA-S as a second real target. The decisive check is the shape of the curves. The claim holds only if two conditions are met. First, the argmax of $A_{real}(s)$ must lie at a coarser scale than the argmax of $A_{syn}(s)$, consistently across seeds and targets. Second, $s^*$ chosen by $D(s)$ must recover at least half the mIoU gap between synthetic-validation selection and the oracle. The tiling control must show the same displacement; otherwise the effect is the resolution paradox rather than a domain effect. The data-scaling sweep then decides whether the effect reflects texture or data scarcity. As a secondary check outside segmentation, we train a detection head on frozen features of Sim 10k and test on KITTI. There, Driving in the Matrix reports 0.6803 Easy and 0.4207 Hard mAP for a fully trained Faster R-CNN with 200k frames. We ask whether scale selection lets a frozen extractor approach that with a fraction of the data.