Counting Scene Motifs to Close the Sim-to-Real Content Gap

Models trained on simulated scenes fail on real ones for two reasons. The first is that the pictures look different, which is the appearance gap. The second, which gets less attention, is that the scenes are arranged differently: which objects appear together, how they are stacked, and who sits behind whom. This is the content gap. We propose to measure and close the content gap using a small learned dictionary of recurring scene arrangements. A simulated scene and a real scene each become a bag of motif IDs drawn from this dictionary. Closing the content gap then reduces to a counting problem: estimate how often each motif occurs in unlabeled real data, and resample or regenerate simulated layouts until their motif counts match.

The best current method, Sim2SG, closes the content gap one image at a time. For each real image it predicts a scene graph (a list of objects plus pairwise relations such as "car behind car"), renders a synthetic twin of that graph in a 3D engine inside the training loop, and trains on the twin. That works, but it is expensive. It also copies every prediction error from a noisy real-image parser straight into the training data. Our bet is that matching the distribution of arrangements is enough, and that doing it in a discrete vocabulary makes that distribution cheap to estimate and robust to parser noise.

Core Claim

Relational scene structure can be quantized into a finite codebook of motifs. By "quantized" we mean that each small neighborhood of a scene graph is snapped to its nearest entry in a learned table of prototypes, the vector-quantization step used in graph tokenizers. Once scenes live in that vocabulary, the content gap becomes a gap between two histograms. Snapping to prototypes absorbs much of the error in scene graphs predicted from real images, so the real histogram can be estimated reliably from those noisy predictions. Correcting the simulator's layout distribution to match it should recover most of Sim2SG's content-gap gain without rendering per-image twins. It should also give a scalar that predicts, before any training, how badly a given simulator configuration will transfer.

This differs from the obvious combination at this intersection. That combination would put a vector-quantization bottleneck inside the perception network and hope it filters out rendering artifacts. That idea runs into a tension in the literature. Adversarially Masking Synthetic to Mimic Real shows that sensor-level transfer requires injecting realistic irregularity. Quantization, by contrast, works by discarding fine detail. It is also hard to distinguish from cheaper regularizers such as dropout. We avoid that tension by quantizing only the layout graph, which has no pixels, and by using the codes to steer the simulator rather than to regularize the image features. The appearance gap is left to the existing adversarial alignment.

Research Program

The method borrows its motif codebook from two sources. GFT unrolls message-passing computation trees and vector-quantizes them into a transferable tree vocabulary, and reports that this reduces negative transfer across unrelated graph domains. Discrete Graph Auto-Encoder (DGAE) quantizes node embeddings into ordered codes and fits an autoregressive Transformer prior over them. It reports that the discrete bottleneck filters out high-frequency structural noise, and it has public code (yoboget/dgae). Neither was ever used for domain transfer from rendered data. We reuse their machinery for a different purpose. Other graph tokenizers solve related problems with other goals: VQGraph uses codes to distill graph neural networks into small MLPs, and Learning Graph Quantized Tokenizers uses hierarchical residual codes as input for sequence models. Their residual quantization is a candidate for a coarse-to-fine version of our vocabulary.

The first stage builds the vocabulary. We train a graph encoder with a quantization layer, where each node's two-hop computation tree is snapped to one of $K$ codes. Training uses ground-truth synthetic scene graphs from Sim2SG's procedural generators, and the training objective is to reconstruct the tree. Nodes carry object class and coarse geometric attributes, and edges carry relation types. DGAE flags that 1-WL message passing cannot tell some substructures apart. Following DGAE, we add path features so that such motifs are not merged into one code.

The second stage estimates the real motif distribution. Sim2SG's current scene-graph predictor parses unlabeled real images. The frozen encoder maps each predicted graph to codes, and pooling gives a real histogram $q$. The simulator's current layouts give a synthetic histogram $p$.

The third stage corrects the simulator in two ways, which we compare against each other. The cheap way reweights existing synthetic scenes. A scene $s$ whose nodes carry codes $c_1,\dots,c_n$ receives the training weight

$$w(s) = \exp\Big(\frac{1}{n}\sum_{i=1}^{n}\log\frac{q_{c_i}+\epsilon}{p_{c_i}+\epsilon}\Big),$$

where $\epsilon$ is a small smoothing constant. The weight is a per-node average of log density ratios, clipped for stability. The generative way fits a DGAE-style autoregressive prior to the real code sequences and samples new layouts from it. Each sampled layout is decoded into objects and discrete relations and placed by the procedural generator's constraint solver. The resulting layouts are rendered offline in one batch rather than inside the training loop.

Appearance alignment stays exactly as in Sim2SG, using a gradient reversal layer, so any gain can be attributed to the content correction. The loop can iterate: a better predictor gives a cleaner $q$, which gives better layouts.

The same histograms serve as a diagnostic. The divergence $\mathrm{KL}(q\,\|\,p)$ measures content mismatch, and the analysis asks whether it predicts downstream transfer accuracy across simulator configurations.

Baselines

The naive reference point is standard domain randomization (SDR), as run in Sim2SG. Layouts, lighting and textures are randomized with no knowledge of the real distribution. Sim2SG reports SDR at Recall@20 0.356 on CLEVR, mAP 0.584 on Dining-Sim, and mAP 0.234 / Recall@50 0.070 on the KITTI "Hard" setting. Any content correction should clear these numbers easily. If it does not, the method has failed outright.

The strongest competitor is Sim2SG itself, with its per-image rendered twins and gradient-reversal alignment. It reports Recall@20 0.888 on CLEVR, mAP 0.729 on Dining-Sim, and mAP 0.316 / Recall@50 0.139 on KITTI Hard. It is the only method in the file that closes the content gap explicitly. We will run it from the same initialization, with the same backbone, scene-graph head and gradient-reversal settings, and with the same count of distinct real images used for pseudo-labeling. We will match total synthetic training images, and we will report rendering compute separately. "Beating" Sim2SG has two senses. The first is exceeding its accuracy at equal rendered-image count. The second, which is the main claim, is matching it within noise at a fraction of the rendering cost. Matching Sim2SG only when we render as many images in a per-image loop would be a negative result for the proposal.

Two ablations isolate the codebook. The first replaces motif codes with raw pseudo-label counts: histograms over predicted object classes and over (subject, predicate, object) triplets, with the reweighting and resampling unchanged. The second replaces the discrete histogram with a continuous density ratio, obtained from a domain classifier trained on unquantized graph-network embeddings of the same predicted graphs. Both ablations get the same encoder architecture, the same tuning budget over their smoothing or clipping constants, and the same number of reweighting iterations. The proposal wins only if motif codes beat both.

Why not just…?

Why not just match label and triplet frequencies instead of learning motifs? Correcting for shifts in how often each label appears is simple and may capture much of the content gap on CLEVR, where scenes are nearly bags of primitives. We treat this as a real threat. Motifs should matter where arrangement is higher-order, such as stacked tableware in Dining-Sim or queued vehicles in KITTI. There, identical triplet counts can hide different configurations. The triplet-histogram ablation settles this. If it matches the codebook on Dining-Sim and KITTI, the discrete vocabulary adds nothing and we would report that.

Why not just use a continuous domain classifier on graph embeddings for importance weights? This is the continuous analog of our method. Because it sees only predicted layouts and never pixels, the image-artifact shortcut problem described in Adversarially Masking Synthetic to Mimic Real does not apply to it directly. Our argument is narrower. With noisy predicted graphs, a continuous classifier can separate the domains using parser errors that never occur in clean synthetic ground truth, and it will then assign extreme weights. Snapping to prototypes should absorb small errors. The file has no direct evidence for this in scene graphs; DGAE's noise-filtering claim concerns reconstruction, not transfer. The controlled parser-noise experiment below tests it directly.

Why not just keep Sim2SG's rendered twins? Twins match each real image exactly, and that may be worth the cost. The case against them is practical: rendering sits inside the training loop, and each twin inherits every error of the parser that produced it. If distribution-level matching falls well short of per-image twins, the conclusion is that the content gap is instance-specific rather than distributional. That would itself be a useful finding.

Why not just quantize the perception features to block shortcuts? This option runs into the tension already described: quantization discards the sensor irregularity that ASM found necessary to inject. It is also hard to separate from generic feature noise. We include it as a side arm, a vector-quantization bottleneck on backbone features with ordinary gradient-reversal training, but it is not the proposal's claim.

Proposed Experiments

The main experiments run on Sim2SG's three settings, CLEVR, Dining-Sim and KITTI, with its metrics: Recall@20 and mAP on the synthetic benchmarks, and mAP and Recall@50 on KITTI Hard. For each method we report mean and spread over three seeds, and rendering GPU-hours alongside accuracy.

The first decisive check is the accuracy–cost curve against Sim2SG. We vary the number of rendered images for twins, reweighted scenes and prior-sampled scenes. The question is whether code-matched layouts reach Sim2SG's 0.316 KITTI-Hard mAP with fewer rendered images and no in-loop rendering.

The second check tests robustness to parser noise. The CLEVR and Dining-Sim targets have ground-truth graphs, so we can corrupt the target graphs at controlled rates by flipping relations, deleting objects and adding false objects. For each rate we measure how far the estimated histogram drifts from the one computed on clean graphs. We do this for motif codes, triplet histograms and continuous-classifier weights, and then measure downstream accuracy. The claim fails if the codes drift as much as the triplet histograms.

The third check tests whether the gap measure is predictive. Across roughly a dozen simulator configurations, varying layout priors, object density and clutter, we correlate $\mathrm{KL}(q\,\|\,p)$, computed before training, with the final target mAP after SDR training. We compare this against the same correlation for a triplet-level divergence.

The fourth check sweeps codebook size $K$ and tree depth. The aim is to find where the vocabulary becomes too coarse to separate arrangements, and where it becomes too sparse to estimate from a few thousand real images.

If code matching recovers most of Sim2SG's content gain at low rendering cost, and beats both ablations on Dining-Sim and KITTI, the discrete vocabulary has earned its place.