Is the Sim-to-Real Gap a Direction in Frozen Embedding Space?

Models trained on rendered or simulated images often fail on real photographs. The standard example is the VisDA benchmark, where a network scores 99% on synthetic test renders and 28% on real images. This note asks a narrow question: if we stop training the image encoder and instead use a frozen foundation model (a large network pre-trained on web-scale images and captions, kept fixed and used only to produce feature vectors), how much of that gap is left, and what shape does it have? Our hypothesis is that, in the frozen embedding space of a model like CLIP, the difference between "rendered" and "real" behaves like the known gap between text and image embeddings. It would be mostly a shared shift along a few directions, roughly the same for every object class. If so, a text-to-image or render-to-photo retrieval system could be trained entirely on synthetic data plus a pile of unlabeled real photos. The recipe would be to estimate those directions, project them out, and fit a small retrieval head. No generative translation and no encoder fine-tuning would be needed.

This matters most for binary hashing. Hashing compresses embeddings into short bit strings so that search reduces to cheap bit comparisons. A shared shift is especially harmful there, because it pushes many coordinates across the zero threshold at once and flips the same bits for every real image. The proposal therefore makes two claims. The first is diagnostic: the frozen-space sim-to-real gap is low-rank and class-shared. The second is practical: removing that structure, while protecting the directions that carry class meaning, closes most of the retrieval gap for both continuous and hashed search. Both claims can be falsified cheaply, and the simplest competing explanation, "just subtract the mean," is built into the design as an ablation rather than argued around.

Core Claim

Let $f$ be a frozen image encoder and $g$ its paired frozen text encoder. We have labeled synthetic images $x_s$ and unlabeled real images $x_r$. We estimate a rank-$k$ "rendering subspace" with orthonormal basis $U$ from the two embedding clouds and replace every image embedding by $P f(x)$, where

$$P = I - UU^\top, \qquad U^\top T = 0 .$$

Here $T$ is the matrix of text embeddings $g(\cdot)$ for the class names or descriptions, so the removed directions cannot carry label semantics. $U$ is chosen as the $k$ directions along which a linear classifier best separates synthetic from real embeddings, subject to that orthogonality constraint.

A retrieval head is then trained on $P f(x_s)$ against $g$ using synthetic labels only, followed by a sign-quantization layer for hashing. Before binarization, codes are centered with the target-domain mean. The method is new because of what it asserts and tests: that a closed-form, text-protected, low-rank correction on frozen features is enough to carry cross-modal retrieval, including its hashed form, from simulator to reality.

Relation to Prior Work

The closest work is GMAIL. It treats diffusion-generated images as a separate modality: it clones CLIP's image encoder, fine-tunes the clone with LoRA (low-rank adapter weights) on up to 12M captioned generated images, and keeps the real encoder frozen as an anchor. It reports COCO text-to-image R@1 rising from 32.7 to 37.5. GMAIL motivates us, since it shows that generated and real images occupy distinguishable regions and that naive mixing hurts. Our proposal differs in four ways. We never update an encoder. We need no captions on synthetic data and no paired real-domain training. We target simulator and CAD renders rather than diffusion outputs. We test whether a fixed linear projection already captures what GMAIL's adapter learns.

Adaptive Cross-Modal Prototypes supplies a principle we reuse: frozen uni-modal cluster centers act as anchors against semantic drift in cross-domain visual-text retrieval, reaching 16.6 R@1 on MSRVTT→MSVD. It studies only natural shifts and trains a prototype-based model, whereas we study synthetic-to-real shifts with a closed-form correction. MUDG shows that CLIP's text-image gap is structured enough that storing centroids in text space fixes retrieval recall. That is the analogy behind our hypothesis, but MUDG uses retrieval only to curate classifier data and never touches rendered imagery.

PRPL showed that a well-chosen frozen backbone plus pseudo-labeling matches end-to-end domain adaptation at 18 to 21 times lower cost (81.2% on Office-Home). We borrow its frozen-feature stance and its pseudo-labeling as a baseline, but PRPL is single-modality classification on natural shifts. On the other side of the tension, Deep Cross-Modal Hashing and Cross-Modal Center Loss argue that end-to-end training beats offline features for hashing and 3D retrieval. Their evidence predates modern foundation encoders and never involved a real test domain; CMCL trains and tests on clean ModelNet CAD data. SynCDR uses image translation and diffusion editing to create synthetic cross-domain pairs, but it depends on text prompts describing the target domain and addresses image-to-image retrieval only. VisDA supplies the benchmark and the size of the problem.

Research Program

The first stage is diagnosis. On VisDA-C (about 152k 3D CAD renders in 12 classes; real COCO validation and YouTube-BoundingBoxes test images), we embed everything with several frozen encoders. We use two or more CLIP-style models, plus an ImageNet EfficientNet for the image-to-image track, following PRPL's finding that backbone choice matters.

Using real labels for analysis only, we compute each class's real-minus-synthetic centroid difference. From these we measure three things. The first is the cosine similarity of these difference vectors across classes; high similarity means the gap is class-shared. The second is the fraction of total synthetic-real separation, measured by linear domain-classifier margin or energy distance, that is captured by the top-$k$ shared directions. The third is how much the difference vectors overlap the span of the class text embeddings.

The hypothesis fails if the difference vectors are close to orthogonal across classes, or if dozens of directions are needed. In that case we report the negative result and fall back to class-conditional alignment.

The second stage is the correction. We estimate $U$ from a disjoint unlabeled half of the real split. One confound needs care: the class mix of the real split differs from that of the renders, so a raw mean difference mixes rendering effects with label-frequency effects. We therefore reweight the synthetic set by class frequencies estimated from zero-shot CLIP predictions on the real images before estimating $U$. The text-orthogonality constraint is the second safeguard.

The third stage is retrieval. We train a small projection head and a 16/32/64/128-bit hashing layer on corrected synthetic embeddings. Two tasks are evaluated on the held-out real split: text query to real-image gallery, and synthetic render query to real-image gallery. We add Office-Home, whose Product and Clipart domains are rendered-looking, as a natural-shift control. MIRFLICKR-25K serves as an in-distribution sanity check that the head and hash layer are competitive when no shift exists.

Baselines

The naive reference point is zero-shot retrieval with the same frozen encoder. Text queries are matched by cosine similarity to raw real-image embeddings, and hash codes are produced by sign quantization with source-mean centering. This baseline uses no synthetic data. A second naive point trains the same head on uncorrected synthetic embeddings, which isolates how much training on renders helps or hurts before any correction.

The strongest existing competitor is GMAIL-style adaptation. We clone the image encoder and apply LoRA on the VisDA renders with a contrastive loss toward class-description text, keeping the original encoder frozen as the real-image branch. It is the strongest competitor because it is the only published method that treats synthetic imagery as its own modality and aligns it to real and text embeddings. We run it on the same synthetic images and unlabeled real images, at matched GPU-hours, with the same hyperparameter-search budget as our head. The only reported numbers for it are COCO R@1 37.5 against CLIP's 32.7 in GMAIL, which is a different, non-sim-to-real setting. No reported result was found on Lacuna for text-to-image or render-to-real retrieval on VisDA-C by any method.

The reference points on VisDA are classification numbers from VisDA. Source-only AlexNet reached 28%, DAN reached 51.62%, and the challenge winner reached 92.8% accuracy. As a transfer check, we will also report nearest-class-text classification accuracy so readers can place our system relative to these.

The second strong competitor is PRPL-style recurrent pseudo-labeling with MMD (a kernel statistic comparing two distributions) alignment, run on the same frozen features, with the resulting classifier's logits used as retrieval scores. For hashing, we train a DEMO-style head on synthetic data, using energy-distance similarity and bidirectional consistency. DEMO reports a MIRFLICKR-25K 128-bit mAP of 0.743 (image-to-text) and 0.728 (text-to-image), which serves as our in-distribution calibration.

The ablations isolate the key ingredients. We sweep the rank: $k=0$ is no correction, $k=1$ is pure mean subtraction, and $k>1$ is the full method. We run the method with and without text-orthogonality, and with and without label-frequency reweighting. We compare target-mean against source-mean centering before binarization.

Beating the baselines means a statistically reliable mAP gain, over three seeds and on both real splits, over zero-shot and over matched-budget LoRA. It also means that $k>1$ with text protection beats $k=1$. If it does not, the simpler method wins and we say so.

Why not just…?

Why not just use zero-shot CLIP with no training at all? For text-to-image retrieval this is a genuine threat, because CLIP never saw the renders and may already retrieve real COCO objects well by class name. Evidence against it being sufficient is indirect. GMAIL and MUDG both find structured adaptation beating zero-shot CLIP, but on natural data. The zero-shot baseline settles this directly. If it is within noise of our method on text queries, the contribution narrows to the render-query and hashing tracks, where zero-shot has no synthetic supervision to exploit.

Why not just subtract the synthetic-minus-real mean? This is the most likely way the idea collapses, and it is included as the $k=1$ ablation. The a priori reason it may fall short is that the raw mean difference mixes rendering style with class-frequency differences between splits, so subtracting it can erase semantic signal. The reweighted, text-orthogonal variant tests exactly this. If plain mean subtraction matches the full method, the paper's finding becomes "one vector suffices," which is still worth reporting and simpler to deploy.

Why not just fine-tune the encoder with LoRA, as GMAIL does? The simulator literature, summarized by VisDA, predicts that updating visual filters on renders invites learning rendering shortcuts. The frozen-feature results of PRPL suggest freezing costs little. Neither result is direct evidence for cross-modal sim-to-real retrieval. The matched-budget comparison is the test, and we also report the gap between synthetic and real accuracy for each method as a shortcut indicator.

Why not just translate renders into photos with a diffusion or translation model? SynCDR reports that prompt-driven editing loses semantic identity at high edit strengths and that translation models are costly to train. GMAIL needed 109 GPU days to synthesize its data. We do not rerun translation at scale. We do report our correction's total cost, which is minutes of embedding plus one eigendecomposition, as the relevant comparison.

Decisive Checks

Three results decide the project. The first is the diagnostic: whether per-class sim-to-real difference vectors share direction strongly enough that a small $k$ explains most of the domain separability. The second is the rank ablation on VisDA-C real-gallery mAP: whether text-protected $k>1$ beats both $k=0$ and $k=1$ in continuous retrieval and at every hash length. The third is the matched-budget comparison against LoRA adaptation. If all three come out in favor, we have a cheap, training-free recipe for simulator-trained cross-modal search. If the diagnostic fails, we have clean evidence that frozen encoders do not flatten the rendering gap into a simple offset, which directly informs the frozen-versus-fine-tuned debate between PRPL and DCMH/CMCL.