Models trained on rendered images, such as CAD models of cars or frames from a video game, lose most of their accuracy on real photographs. On the VisDA benchmark, a classifier that is 99% accurate on synthetic images drops to 28% on real COCO images (VisDA). The usual explanation is that the network learns rendering shortcuts, such as clean lighting or missing clutter. This proposal looks at a less studied symptom of those shortcuts: the network also learns the wrong map of which classes resemble which. If flat CAD shading is enough to tell a bus from a truck, the network has no reason to keep "bus" close to "truck" and far from "horse." When real images arrive and the shortcut vanishes, the mis-learned map turns into specific, predictable confusions between particular pairs of classes.
The idea is to pin down that map using a source that never saw a rendered pixel. We build a class-to-class similarity matrix from language: attribute descriptions of each class embedded by a text encoder, plus word embeddings. During synthetic training, we require the network's own class-to-class similarities to match that matrix, while leaving each class free to keep several internal clusters. Put briefly, the simulator teaches the network what each class looks like, and language tells it how the classes relate to each other. A wrong-looking simulator then cannot quietly redraw that geometry.
Core Claim
The proposal makes two claims, and each can be refuted on its own. The first is diagnostic. Synthetic training distorts the inter-class similarity structure, meaning the pattern of similarities between class centers. That distortion can be measured per class pair, and for a given pair it predicts how often the two classes are confused on real data. The second is corrective. Matching the network's class similarity matrix to a language-derived matrix improves real-domain accuracy beyond what per-sample distillation from a frozen natural-image model already gives. It also beats a structure-free "spread the classes apart" regularizer. The comparison against a structure-free regularizer is the important one. Several existing methods keep classes separated, but none in the literature we reviewed supplies the correct relative distances when the source domain is synthetic.
Relation to Prior Work
Prior work on keeping inter-class structure consistent motivates this proposal, but every version assumes a trustworthy anchor. Topological Consistency Adaptation (Ni et al.) preserves the source model's class distances during continual test-time adaptation, using a uniformity loss and a compactness loss. Its authors note that it degrades when the source topology is poor, and it was evaluated only on synthetic corruptions such as CIFAR-10-C and ImageNet-C. In our setting the synthetic source is the poor anchor, so preserving it would preserve the error. Sample Consistency Alignment (Wei et al.) keeps a similarity hierarchy across art, photo and sketch domains using only visual batches, with no external anchor. FedMAP (Yipan Wei et al.) is closest in mechanism: it forces visual centroid similarities to mirror a text relation matrix derived from GPT-4o attribute codebooks. FedMAP works in federated medical prompt tuning and warns that hallucinated descriptions distort the result. We reuse its text-relation-matrix technique in a different regime, where the visual side is corrupted by a simulator rather than by client heterogeneity, and we test explicitly against hallucination. VS-Boost's relational metric learning (Li et al.) and MASK's word-embedding bridge (MASK) show that semantic relations can shape visual geometry for zero-shot recognition on natural photos. VS-Boost also reports that the benefit weakens when the semantic space is low-dimensional, as on SUN. That is a real constraint for us, since VisDA has only 12 classes.
On the synthetic-to-real side, the closest work is Automated Synthetic-to-Real Generalization (ASG). ASG penalizes per-sample KL divergence between the trained model and a frozen ImageNet model to stop the network forgetting real-world features. ASG constrains each image's output distribution. It does not constrain the geometry among class centers, and its reference is itself a visual model that sees the same synthetic images. Distribution Regularized Self-Supervised Learning (DRSL) creates a tension we must resolve. It argues that forcing each class into one tight cluster discards the sub-modes, for example different vehicle types within "car," that are needed to transfer from GTA-V to Cityscapes. It therefore models each class as a mixture, but it never constrains relations between classes. Minimum Class Confusion (MCC) penalizes confusion on target predictions without any notion of which classes ought to be near each other. Finally, pixel-level translation (PixelDA) acknowledges that it handles only low-level appearance shifts, not the structural differences between CAD geometry and real objects that our constraint targets.
Research Program
The work proceeds in three parts.
Measuring the distortion. We train source-only models on VisDA-C synthetic images. We then compute class-center similarity matrices on held-out synthetic data and on the labeled real validation set. From these we measure the per-pair distortion, which is the difference between the two matrices, and correlate it with the per-pair confusion rate on real data. Labeled real data are used only for this analysis and for an oracle condition, never for training the proposed method.
Building the anchor. For each class we generate attribute descriptions with a language model, embed them with a text encoder, and average them. We also build a second, independent anchor from word embeddings. Comparing the two lets us detect when the language model has hallucinated attributes.
Training with the anchor. The training loss lets each class keep its internal sub-modes. Let class $c$ be represented by $K$ mode vectors $m_c^1,\dots,m_c^K$, estimated with a moving average over mini-batches as in DRSL. The model's relation between classes $i$ and $j$ is the average cosine similarity across their modes,
$$S_{ij}=\frac{1}{K^2}\sum_{a,b}\cos(m_i^a,m_j^b).$$
Let $A_{ij}$ be the anchor similarity from language. Let $\sigma_\tau(\cdot)_i$ denote a softmax over the off-diagonal entries of row $i$ at temperature $\tau$. The added loss compares rows of the two matrices as distributions:
$$L_{rel}=\sum_i \mathrm{KL}\big(\sigma_\tau(A)_i \,\|\, \sigma_\tau(S)_i\big).$$
The total objective is synthetic cross-entropy plus $\lambda L_{rel}$. Comparing rows as distributions makes the loss depend on relative ordering rather than absolute scale, and the language and visual similarity scales are not comparable. We deliberately add no compactness term, so the DRSL concern is addressed by design. An ablation that adds TCA's compactness term tests whether that choice matters.
In the unsupervised adaptation setting, where unlabeled real images are available, the same loss is also applied to target class centers computed from confident pseudo-labels. Here the anchor serves a second purpose: it stops self-training from collapsing nearby classes into each other.
Baselines
Naive reference. The do-nothing reference is source-only training on synthetic data. ASG reports 28.2% on VisDA-17 under a large learning rate (ASG), and VisDA reports 28% for AlexNet (VisDA).
Main competitor when no real data is used. The strongest competitor in this setting is ASG's proxy-guidance KL loss, which ASG reports at 58.7% on VisDA-17. Full ASG, with learning rates tuned by reinforcement learning, reaches 61.1%. Proxy guidance is the most direct rival because it also injects knowledge of natural images into synthetic training, only per sample rather than through class geometry. We will rerun proxy guidance ourselves on the same backbone ASG reports, with the same synthetic data, epochs, and an equal hyperparameter search budget, and with three seeds. Beating it means a statistically reliable gain from adding $L_{rel}$ on top of proxy guidance. We do not count a gain that appears only when $L_{rel}$ replaces proxy guidance as winning. The learning-rate policy in full ASG is orthogonal to our method, so we compare against its reported 61.1% for reference and do not claim to beat it unless we do.
Ablations that isolate the anchor. Three ablations isolate what the anchor contributes:
- A uniform anchor, with all off-diagonal entries equal, is effectively TCA's uniformity idea. It tests whether any gain comes from simply spreading classes apart.
- A shuffled anchor permutes the language matrix. It tests whether any gain comes from generic regularization.
- A frozen-visual anchor computes $A$ from an ImageNet-pretrained backbone applied to synthetic images. It tests whether language is needed or whether any pretrained relation would do.
An oracle anchor, computed from labeled real images, gives the upper bound on what relational correction can deliver at all.
Adaptation setting. Here the references are DAN at 51.62% and the challenge winner's pseudo-labeling pipeline at 92.8% (VisDA), and ASG used as initialization at 84.6%. We run MCC with and without $L_{rel}$ under identical settings; no reported VisDA number for MCC was found on Lacuna, so MCC is rerun rather than quoted.
Segmentation. For GTA5→Cityscapes, ASG reports FCN-Res50 improving from 22.17% to 29.64% mIoU without target data. DRSL reports 47.8% mIoU with adaptation, and 46.7% on SYNTHIA→Cityscapes. We add $L_{rel}$ to each setting. Success means improving the matching baseline under the same protocol.
Why not just…?
Why not just use ASG's proxy guidance, since a frozen ImageNet model already carries real-world class structure? This is the most serious threat. Per-sample KL toward a natural-image model may already implicitly preserve class geometry. ASG's own analysis states that it relies on ImageNet covering the target classes, and it never measures geometry. Two experiments settle the question. The first stacks $L_{rel}$ on top of proxy guidance; if proxy guidance already fixes the geometry, the gain will vanish. The second measures the class-geometry distortion under proxy guidance alone. If proxy guidance closes the distortion and the stacked gain is zero, we will report that relational anchoring is redundant with it.
Why not just spread the classes uniformly apart, as TCA does? TCA's uniformity loss helps on corruption benchmarks (Ni et al.). However, uniform spacing treats bus–truck the same as bus–horse, so it cannot encode which pairs a simulator wrongly pulled apart. The uniform-anchor ablation tests this directly. If uniform spacing matches the language anchor, our core claim fails.
Why not just minimize target confusion with MCC? MCC is cheap and effective (MCC), but it requires target data. It also pushes all confused pairs apart equally, including pairs that should stay close. In the adaptation setting we expect $L_{rel}$ to be complementary to MCC. The MCC-plus-anchor run tests this claim, and a per-pair analysis checks whether MCC over-separates semantically close classes.
Why not just translate synthetic images to look real? PixelDA reports that pixel translation addresses only low-level appearance, not geometric differences (PixelDA). It is also orthogonal to our loss. We treat it as out of scope rather than as a competitor, and say so.
Decisive Checks
The idea lives or dies on three results. First, the oracle anchor must improve real accuracy over source-only training; if even correct real-world geometry does not help, the premise is false and we stop. Second, per-pair distortion must correlate with per-pair real confusion. Third, the language anchor must beat both the shuffled and uniform anchors and must add to proxy guidance.
Two secondary checks follow. Agreement between the language-model anchor and the word-embedding anchor should predict which classes benefit, which tests FedMAP's hallucination warning. With only 12 VisDA classes, the matrix has 66 free entries, so the constraint may be too weak to matter. GTA5→Cityscapes, with 19 classes, is the test of whether the effect grows with richer relational structure.
References
- VisDA: The Visual Domain Adaptation Challenge
- Maintaining Consistent Inter-Class Topology in Continual Test-Time Adaptation
- Indirect Alignment and Relationship Preservation for Domain Generalization
- Rethinking Federated Prompt Learning for Medical Images: From Textual Tuning to Visual Manifold Anchoring
- VS-Boost: Boosting Visual-Semantic Association for Generalized Zero-Shot Learning
- Multimodal Aligned Semantic Knowledge for Unpaired Image-text Matching
- Automated Synthetic-to-Real Generalization
- Distribution regularized self-supervised learning for domain adaptation of semantic segmentation
- Minimum Class Confusion for Versatile Domain Adaptation
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks