Which Codewords Survive the Jump from Simulator to Reality?

A model trained on synthetic images, such as game-engine frames or rendered CAD parts, often collapses on real photos. One promising fix is to force the network's internal features through a small "codebook": a learned list of discrete feature prototypes, where every image patch must be replaced by its nearest entry. A small codebook cannot store the fine rendering quirks that the network would otherwise memorize. The open question is how small it should be. Across many fields, growing a dictionary helps at first, then levels off, then hurts. We propose that under synthetic-to-real shift the harmful regime arrives much earlier than on in-domain data, because the extra codewords are spent on simulator artifacts.

We also propose a way to find those codewords using only synthetic data. Perturb an image's frequency amplitudes while keeping its phase, which leaves object structure intact, and check which codewords the patches still map to. Codewords whose assignments flip under this perturbation encode rendering texture rather than content. Counting the stable codewords should predict the codebook size that transfers best to real data, and removing the unstable ones should recover transfer accuracy without any real labels.

Core Claim

Each strand of prior work supplies a piece. Domain Generalization via Discrete Codebook Learning (DDG) quantizes feature patches to a fixed set of 256 codewords and proves that quantization bounds the Wasserstein distance between domains. It reports 84.48% on PACS. However, it never varies the codebook size and never tests synthetic-to-real shift. This paper supplies our method and motivates our question.

Dictionary-scaling work supplies the expected curve shape. On the Role of Seed Lexicons in Learning Bilingual Word Embeddings finds that adding long-tail translation pairs beyond 5,000–10,000 degrades the mapping, because noisy, poorly estimated entries warp it. ATD reports that oversized token dictionaries spread attention thinly over redundant entries and wash out high-frequency detail. In VQ-MPT, a larger dictionary did not reliably lower reconstruction loss.

PASTA supplies the diagnostic tool. It shows that synthetic images have too little high-frequency amplitude variance compared with real photos, and that perturbing amplitude while preserving phase narrows the gap.

Our claim joins these pieces, and is a measurable claim that could fail. Let $q(x)$ be the codeword assigned to a patch feature of image $x$. Let $T(x)$ be a PASTA-style amplitude perturbation with phase held fixed. We define the instability of codeword $k$ as

$$s_k = \Pr\big[q(T(x)) \neq k \mid q(x) = k\big],$$

estimated over synthetic training patches. We define the effective codebook size $K_{\text{eff}}$ as the number of codewords with $s_k$ below a threshold $\tau$ and usage above a floor $u$. We hypothesize three things.

First, the real-target optimum $K^\ast_{\text{real}}$ lies well below the synthetic-validation optimum.

Second, $K_{\text{eff}}$, computed with no real data at all, tracks $K^\ast_{\text{real}}$ across benchmarks.

Third, take a large codebook and collapse its unstable codewords into their nearest stable neighbours. This should match the best small codebook while keeping the representational room that geometric detail needs.

This last point matters for assemblies. In Towards Sim-to-Real Industrial Parts Classification with Synthetic Dataset, accuracy fell from 80–90% on isolated parts to about 40–50% on assembled parts. There, a uniformly tiny codebook may erase the small sub-components that decide the label.

Research Program

We build on DDG's quantization layer, including its exponential-moving-average codebook updates and teacher-student consistency. We insert it at the same intermediate stage of each backbone.

Step 1: map the scaling curve. We sweep codebook size $K$ from 16 to 65,536 in powers of four. For every $K$, we record three things: accuracy on held-out synthetic data, accuracy on real target data, and the per-codeword instability $s_k$. The perturbation $T$ uses PASTA's frequency-proportional amplitude jitter. We also report band-restricted variants, perturbing only low, mid or high frequency bands, to see which bands produce unstable codewords.

Step 2: a label-free rule for size. We fix $\tau$ and $u$ once, on one benchmark. We then test whether $K_{\text{eff}}$ predicts $K^\ast_{\text{real}}$ on the others without retuning.

Step 3: stability-based pruning. We train with a large $K$. We then reassign every patch mapped to an unstable codeword to its nearest stable codeword. Finally, we fine-tune briefly on synthetic data only.

Step 4: a validity check on the instability score. We use unlabeled real images only for this analysis, never for training. If unstable codewords really encode simulator artifacts, they should be under-used on real images relative to their synthetic usage. Their usage ratio should correlate negatively with $s_k$.

Baselines

The naive reference point is a source-only network with no codebook, trained on the same synthetic data and backbone. Every gain is measured against it. On VisDA-C, VisDA reports that an AlexNet drops from 99% on synthetic data to 28% on real COCO images. On GTAV→Cityscapes, PASTA reports a DeepLabv3+ source-only baseline of 28.95 mIoU.

The strongest existing method for synthetic-to-real generalization in this file is PASTA itself. It reaches 44.12 mIoU on GTAV→Cityscapes with DeepLabv3+ and reports a +16.9 mAP gain on Sim10K→Cityscapes. I found no reported PASTA result on Lacuna for VisDA-C classification, so we will run it ourselves there.

The strongest discrete-codebook competitor is DDG at its default $K=256$. We run it twice: alone, and combined with PASTA augmentation during training.

On SIP-17, the reference is ConvNeXt trained on the domain-randomized split, as reported in the industrial-parts paper: 83.2% on isolated parts and roughly 40% and 50% on engine and wheel assemblies.

The VisDA-C challenge winner reached 92.8%, but it used unlabeled target data through pseudo-labeling. It is therefore not a fair comparison for our target-free setting, and we report it only for context.

All methods share the same backbone, synthetic data, number of training epochs, and number of hyperparameter trials. Every hyperparameter, including our $\tau$, is selected on synthetic validation data only.

We add three ablations to isolate the key ingredient:

  1. Random pruning of the same number of codewords, to show it is the choice of codewords that matters and not just their count.
  2. Pruning by usage frequency alone, which is the obvious long-tail heuristic suggested by the lexicon literature.
  3. An oracle that picks $K$ using real-target labels. This is the ceiling a label-free rule can approach.

We count it as success only if all three of the following hold. The pruned large codebook must beat both DDG+PASTA and PASTA alone on at least three of the four benchmarks. Its result must fall within the seed-to-seed spread of the oracle-chosen $K$. And it must beat both random and frequency pruning.

Why not just…?

Why not just tune $K$ on synthetic validation data? The synthetic validation set shares the simulator's artifacts, so it rewards exactly the codewords we suspect are harmful. Lexicon studies and ATD show that held-out in-domain performance often plateaus rather than drops, which would push the chosen $K$ upward. The scaling sweep tests this directly. If the synthetic-validation optimum coincides with $K^\ast_{\text{real}}$, our first hypothesis fails, and the size-selection rule becomes unnecessary.

Why not just apply PASTA and skip the codebook? This is the most serious threat. Augmentation might already stop the network from relying on amplitude artifacts. However, augmentation acts on inputs and leaves capacity unconstrained. DDG reports that continuous features remain prone to memorizing domain nuances, and pixel-level randomization did not escape the albedo trap on assemblies in the industrial-parts study. The DDG+PASTA arm settles this. If PASTA alone matches it, and pruning adds nothing on top, we will report that the codebook is redundant once amplitude is augmented.

Why not just use a continuous low-dimensional bottleneck? The point of a codebook is a hard, inspectable capacity limit. A continuous projection to dimension $d$ is the natural rival. We include a matched linear bottleneck whose $d$ is swept over the same range. If it reproduces the same rise-then-fall curve and the same optimum, the discrete part is not essential. Even then, the per-codeword instability diagnostic has no continuous counterpart.

Why not just render more synthetic data? Driving in the Matrix shows gains from 10k to 200k GTA V frames: 0.6803 mAP on KITTI Easy, beating a model trained on real Cityscapes data. It also reports a persistent gap. VisDA-C's roughly 152,000 renders still left AlexNet at 28%. We additionally rerun the $K$ sweep at 25%, 50% and 100% of VisDA-C's source set to see whether more data moves $K^\ast_{\text{real}}$. If it does, that would suggest the codebook limit and the data limit trade off against each other.

Proposed Experiments

Benchmarks. We use VisDA-C with ResNet-50 and report mean per-class accuracy. We use GTAV→Cityscapes with DeepLabv3+ and report mIoU. We use Sim10K→Cityscapes car detection and report mAP. We use SIP-17, reporting isolated-part and assembly accuracy separately. As a sanity check, we reproduce DDG's 84.48% on PACS with ResNet-18 before changing anything.

All results use three seeds. The tolerances in the checks below are set before running anything.

Decisive check 1: early peak. On at least three benchmarks, real-target accuracy should peak at a $K$ at least four times smaller than where synthetic accuracy plateaus. Beyond that peak, real accuracy should decline by more than seed noise.

Decisive check 2: $K_{\text{eff}}$ predicts the optimum. With $\tau$ fixed on VisDA-C, the predicted $K_{\text{eff}}$ should land within one sweep step of $K^\ast_{\text{real}}$ on each of the other benchmarks. We also report the rank correlation between $K_{\text{eff}}$ and $K^\ast_{\text{real}}$.

Decisive check 3: instability means synthetic-specific. Across codewords, real-to-synthetic usage ratio should correlate negatively with $s_k$.

Decisive check 4: pruning helps assemblies. On SIP-17 assemblies, the pruned large codebook should beat the best small codebook. That would show that keeping stable fine-grained codewords matters where small parts decide the label. This is where we expect the clearest separation from a uniformly small codebook such as DDG's.

If checks 1 and 2 fail, the proposal reduces to "codebooks plus augmentation." That would still be a useful negative result, but a smaller one.