Training Spatial Readouts in Simulation for Frozen Vision Encoders

Contrastive image-text models such as CLIP often cannot tell "the mug is left of the plate" from "the mug is right of the plate." Two obvious fixes both have a problem. Real images with balanced spatial captions are scarce. Synthetic scenes are cheap and can be balanced perfectly, but models trained on them tend to fail on real photographs.

This note makes a narrower bet that is cheap to test. The pretrained vision encoder already stores spatial layout in its per-patch features, which are the grid of local image descriptors it produces before collapsing them into one summary vector. Suppose we freeze that encoder, which was pretrained on real images, and train only a small component that reads those patch features conditioned on the caption. Then synthetic data only has to teach that small reader what "left of" means in terms of patch positions. It never gets to reshape the visual features themselves. If the bet holds, a few GPU-hours of rendered scenes produce a spatially competent contrastive model that transfers to real photos, and the synthetic-to-real gap becomes a quantity we can measure in one small module.

Core Claim

The claim is conditional and falsifiable. Take a frozen real-pretrained encoder and train a caption-conditioned patch readout on procedurally rendered scenes only. On real spatial benchmarks it will beat three alternatives trained on the same synthetic data: a readout on the pooled image token, a model whose last backbone blocks are fine-tuned, and a readout trained without the caption conditioning. Its remaining error should be mostly explained by how differently relations are encoded in real versus synthetic patch features, and much less by rendering realism.

Two findings motivate this. Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encoder shows that the same frozen CLIP vision encoder supports 93.2% pair accuracy on What'sUp Left/Right when a language model reads all 576 patch tokens, while CLIP's pooled token gets 1.9%. It also shows that removing the question from the contrastive conversion (VLM2Vec) drops accuracy from 95.1% to 0.0%. The spatial information is present in the patches; what fails is the way it is extracted. What's "up" with vision-language models? introduced What'sUp, a set of tripod-captured photographs of household objects in controlled relative positions. It found that fine-tuning on spatial datasets can reduce accuracy on those controlled real scenes, which the authors attribute to dataset-specific shortcuts. That result constrains the design. It suggests the damage comes from updating the visual features, so the backbone must stay frozen.

The closest synthetic work, RLS3, mines hard spatial configurations in Unity with a reinforcement-learning agent and raises CLIP from 50% to 56%. It evaluates only on held-out synthetic scenes, and its authors state that transfer to the real world is untested. This proposal differs in three ways. It puts the learning in a patch-level readout rather than in the whole vision-language model. It evaluates on real photographs. It asks where the transfer gap lives rather than how to mine harder synthetic samples.

Research Program

The model uses frozen encoders plus a trainable readout. The vision side is a frozen CLIP ViT-L/14 at 336 pixels, which produces the same 576 patch tokens studied by Li et al. We also test a frozen DINOv2 encoder paired with a frozen text model through a Token Projector. Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignment introduced this setup: it trains an 11M-parameter projector between frozen backbones and reports that DINOv2 keeps more patch-level detail (31.37 versus 23.46 mIoU on Pascal VOC). We reuse its alignment recipe rather than contribute to it.

The readout is a small cross-attention block of a few million parameters. The caption's token embeddings act as queries over the patch tokens, and the block emits one image embedding that is scored against the caption embedding. It is trained contrastively. Each batch contains hard negatives that differ from the positive caption only in the preposition, so the model cannot succeed by recognizing objects alone.

The synthetic data is intentionally cheap. We render two- and three-object tabletop and floor scenes in Blender from free asset libraries and pair them with templated captions. Every object pair appears in every relation, which removes the "usually on the table" frequency bias that Kamath et al. identify in web captions. Rendering fidelity is a controlled variable with three levels: flat-shaded, textured, and domain-randomized in lighting, texture and camera. This tells us whether realism matters once the backbone is frozen.

The analysis is a gap decomposition. We fit the same readout on real features using an object-disjoint cross-validation split of What'sUp, which gives a real-data oracle. The difference between the oracle and the synthetic-trained readout is the transfer gap. We then test whether that gap is predicted by representational mismatch. For each relation we compute centered kernel alignment (a similarity score between two sets of features, used in Freeze-Align to predict how easily spaces align) between real and synthetic patch features. We also train a linear probe to detect synthetic-versus-real from the readout's output. If the gap tracks the patch-feature mismatch, the bottleneck is the encoder. If it tracks the probe, the readout has learned simulator shortcuts.

Baselines

The naive reference points are zero-shot CLIP ViT-L with its pooled token, reported at 1.9% pair accuracy on What'sUp Left/Right by Li et al., and a random-chance floor. We re-run both under our evaluation code so the numbers are comparable.

The strongest existing methods are LLaVA-1.5-7B at 93.2% and VLM2Vec at 95.1% pair accuracy on the Left/Right split, both from the same paper. These read the same frozen encoder through a 7B language model. We treat them as a ceiling for extraction quality, not as a same-budget competitor, and report GPU-hours and inference cost alongside accuracy. The file reports no LLaVA or VLM2Vec result on the other What'sUp splits, on COCO-spatial or on GQA-spatial, so we will measure those ourselves. The best fine-tuned model in Kamath et al., BLIP fine-tuned on VQAv2, reaches 56% and serves as the prior real-data fine-tuning reference.

The same-budget competitors all see the same synthetic images, the same number of optimizer steps and the same hyperparameter-search budget of twenty trials each. The first is RLS3-style full contrastive fine-tuning of CLIP on our renders, with random sampling since the RL miner is not the variable under study. The second fine-tunes CLIP's last four vision blocks together with the readout. The third is a readout that attends to the pooled token instead of the patches. The fourth ablation isolates the key ingredient: a patch readout with a fixed learned query in place of the caption-conditioned queries.

Beating the baselines means two things. The caption-conditioned patch readout must beat every same-budget competitor on real What'sUp and COCO-spatial by a margin larger than the spread across seeds. It must also recover at least most of the distance between zero-shot CLIP's 1.9% and LLaVA's 93.2% on Left/Right, while using orders of magnitude less inference compute than LLaVA.

Why not just…?

Why not just use LLaVA or VLM2Vec, which already solve Left/Right? This is the real threat, and we take it seriously. If a 7B model is acceptable, the accuracy problem on that split is essentially solved. Our contribution is then cost and modularity. The result is a CLIP-style contrastive model that still indexes and retrieves like CLIP, with a few-million-parameter addition trained on free data. The note is only interesting if the readout gets close to LLaVA at a small fraction of its compute. We report accuracy against compute directly, and if the readout stalls far below LLaVA we will say so.

Why not just fine-tune on real spatial data such as COCO-spatial? Kamath et al. report that spatial fine-tuning reduced accuracy on controlled real scenes, and web data is heavily biased in which relations it shows. We include a readout trained on real COCO-spatial-style data as a direct comparison on What'sUp. If it matches the synthetic readout, the synthetic pipeline is unnecessary and the finding becomes "the readout, not the data, is what matters." That would still be a useful result.

Why not just fine-tune the whole model on synthetic scenes, as RLS3 does? Updating the visual features is exactly how simulator shortcuts get in. The frozen-versus-last-blocks-tuned comparison tests this directly. If partial fine-tuning wins on real data, the core claim is wrong and we will report that.

Why not just make the renders photorealistic or translate them to look real? Pixel-level translation such as PixelDA fixes appearance but not geometry, and spatial relations are geometric. The fidelity ladder tests whether realism matters. If domain-randomized renders substantially beat flat-shaded ones, we will add a translation baseline, and the claim that the frozen backbone makes realism unimportant is weakened.

Decisive Checks

We evaluate on all What'sUp splits, COCO-spatial and GQA-spatial, with three seeds, and report pair and individual accuracy.

The first decisive test is the patch-versus-pooled comparison under synthetic training. If the pooled readout matches the patch readout on real data, the mechanism is wrong. The second is frozen versus partially tuned, which tests the shortcut account. The third is the gap decomposition: whether the oracle-minus-synthetic gap correlates with real-versus-synthetic patch-feature alignment across relations and backbones, and in particular whether CLIP and DINOv2 differ in this. The fourth is whether flat-shaded renders transfer about as well as randomized ones. If so, sim-to-real for spatial language is mostly handled by freezing a real-pretrained encoder, and it can be done on a single GPU.