A vision model trained on rendered images and then shown real photographs usually organizes what it sees by where the image came from rather than what is in it. A real photo of a bicycle ends up nearer to a real photo of a car than to a rendered bicycle. Natural-language processing has seen a close cousin of this failure. Word vectors learned from co-occurrence place "hot" next to "cold" because both fill the same slot in "the water is ___". A cheap post-hoc fix exists for that case: counter-fitting, an optimization run after training that pulls known synonyms together, pushes known antonyms apart, and keeps every vector close to where it started. It runs in minutes and never touches the model's weights (Counter-fitting Word Vectors to Linguistic Constraints). We propose to run the same push-pull optimization on the frozen feature vectors of a vision backbone. The constraints come from the sim-to-real setting itself: synthetic and real images believed to show the same class are pulled together, and same-domain images believed to show different classes are pushed apart.
The setting we care about is common and underserved. The best current answers to the sim-to-real gap all retrain the backbone. They use adversarial discriminators (DIRL), teacher-student consistency training on unlabeled real data (Self-ensembling for visual domain adaptation), or full fine-tuning of a 304M-parameter vision-language encoder (VLTSeg). When the backbone is a frozen API, a certified on-device model, or simply too expensive to retrain, practitioners are left with a linear probe or nothing. We ask how much of the gap a minutes-long, weight-free refinement of the feature space can close. We also ask whether the repulsive half of counter-fitting is what does the work. That half has no counterpart in existing feature-smoothing methods.
Core Claim
Our hypothesis is that domain confounding in frozen features has two parts. The first is a global shift: real features sit in a different region than synthetic ones. Cheap per-domain re-standardization removes most of it. The second part is class-conditional, and it is where the real errors live. After standardization, real images of different classes still form tight clusters together, because they share a real-world "look" the source training never had to separate.
Attraction-only refinement makes this second problem worse. Classical retrofitting (Dodge et al.) and polynomial graph smoothing over nearest-neighbor graphs (More Discriminative Sentence Embeddings via Semantic Graph Smoothing) average each point with its neighbors, so they blend these confused real neighborhoods further. This is the post-hoc analogue of the negative transfer that DIRL documents for global marginal alignment. DIRL fixes it with class-conditional discriminators and a triplet-style separation loss trained end to end. We claim that an explicit repulsion term applied post hoc to frozen features recovers most of that benefit, at a small fraction of the cost.
Research Program
The method has four stages.
Stage zero: standardize. Features from the frozen backbone are standardized separately per domain. We center each domain, whiten it to match second-order statistics as Deep CORAL does (VisDA), and length-normalize. This step addresses the anisotropy and hubness problem that the research file flags as the main obstacle to applying retrofitting to vision embeddings. It is deliberately part of the method, not a competitor. Every claim below is about what is gained on top of it.
Stage one: build constraints. We build two sets of pairs, an attract set $A$ and a repel set $R$, under two supervision regimes.
In the unsupervised regime, a real image is tied to a source class only when two independent signals agree with high confidence. One signal is the synthetic-trained classifier. The other is zero-shot classification by a frozen CLIP text encoder, the language anchor whose sim-to-real invariance VLTSeg reports. For such a real image $i$:
- we add an attract pair from $i$ to its nearest synthetic exemplars of that class;
- we add repel pairs from $i$ to its nearest real neighbors that carry a confident, different label.
Synthetic-synthetic repel pairs come for free from source labels.
In the few-shot regime, the 5–10 labeled real images per class used by DIRL replace the agreement filter for the labeled points.
Stage two: optimize. With original features $x_i$, refined features $z_i$, cosine distance $d$, margin $\delta$, and weights $\lambda_R, \lambda_P$, we minimize
$$ \mathcal{L}(z) = \sum_{(i,j)\in A} d(z_i,z_j) \;+\; \lambda_R \sum_{(i,j)\in R} \max\big(0,\ \delta - d(z_i,z_j)\big) \;+\; \lambda_P \sum_i d(z_i,x_i). $$
The three terms are the synonym-attraction, antonym-repulsion, and vector-space-preservation terms of counter-fitting. The optimization is plain stochastic gradient descent over the vectors, not the network. Classification then uses nearest class prototype, or label propagation, in the refined space.
Stage three: make it inductive. Classical retrofitting only moves points that appear in the graph, so an unseen image would get no benefit. We therefore fit a linear map from $x$ to $z$ by weighted least squares, borrowing the solver that Model Fusion via Retrofitting uses to align representations without retraining. That paper fuses networks rather than crossing domains; we use only its post-hoc least-squares machinery. Inductive use lets the refinement fit on one real split and apply to another.
Analysis. Beyond accuracy, we measure neighbor purity for real images before and after each stage. This is the fraction of a real image's $k$ nearest neighbors that share its class, compared with the fraction that merely share its domain. If the hypothesis is right, standardization should raise cross-domain neighbor rates. Repulsion, and only repulsion, should raise same-class purity among real neighbors.
Baselines
Naive reference points. The do-nothing reference is the frozen synthetic-trained backbone with its own classifier head, evaluated on real images. For calibration, the VisDA challenge paper reports an AlexNet dropping "from 99% accuracy on synthetic data to just 28% on real COCO images without adaptation" (VisDA). The second reference is per-domain standardization plus a nearest-prototype classifier with no refinement, which isolates stage zero.
Strongest post-hoc competitor. This is attraction-only refinement under the same stage-zero preprocessing. We test it in two forms: the polynomial graph filters of Semantic Graph Smoothing on a joint nearest-neighbor graph, and classical retrofitting using our exact attract set $A$ with $\lambda_R=0$. Both are cheap, need no network gradients, and share every input with our method except the repulsive term. Because they differ from our method only in that term, this baseline is also the key ablation. No sim-to-real number exists for it; there is no reported result found on Lacuna for graph smoothing on vision features. Beating it means a consistent, seed-robust gain from $\lambda_R>0$ on every benchmark, together with the predicted neighbor-purity signature.
Strongest weight-updating competitors. In the unsupervised regime this is self-ensembling. The paper reports "92.8% test accuracy" on VisDA-2017 classification (Self-ensembling). In the few-shot regime it is DIRL, reported at "83.7%" on MNIST→SVHN with 10 labels per class (DIRL). The published figures used their own architectures and engineering, so we cite them as reference ceilings and also rerun both methods on our backbone, with the same tuning budget in hyperparameter trials. We report wall-clock and memory alongside accuracy.
We do not expect to beat 92.8%. Success means recovering most of the gap between source-only and backbone-matched self-ensembling, at a small fraction of its compute. The few-shot regime is where we do aim to match DIRL. For the frozen-versus-fine-tuned question in segmentation, the ceiling is VLTSeg's "63.1% DG mean across four real-world datasets" from GTA5.
Why not just…?
Why not just fine-tune the backbone, as VLTSeg does? VLTSeg explicitly reports that freezing vision-language backbones is suboptimal and that full fine-tuning is needed for top domain-generalization numbers. That directly contradicts the optimism of the post-hoc literature, and we take it seriously. Our claim is not parity. It is that when weights cannot move, repulsive refinement is the best available option, and that it narrows a measurable share of the gap. The segmentation extension reports frozen-plus-refinement against VLTSeg's fine-tuned number on the same benchmark, so readers can see the price of not fine-tuning directly.
Why not just re-standardize each domain and use prototypes? This is the alternative most likely to explain away the gains, which is why we absorbed it into the method as stage zero rather than arguing against it. The experiment that settles it is the ablation chain: raw features, then standardized, then standardized plus attraction, then standardized plus push-pull. If the last step adds nothing, the core claim is false.
Why not just run graph smoothing or label propagation on a nearest-neighbor graph? The research file itself notes that smoothing relies on similarities in a space where shortcut features dominate. We predict it will lower same-class purity among real neighbors while raising cross-domain mixing. The neighbor-purity analysis and the $\lambda_R=0$ ablation test that prediction directly.
Why not just train a linear probe on the few real labels? With 5–10 labels per class, a probe is a real threat in the few-shot regime. We include it as a baseline there, and we also combine it with our refined features. If push-pull refinement does not improve the probe's accuracy, the method only helps in the unsupervised regime, and we would say so.
Proposed Experiments
The primary benchmark is VisDA-C: about 152k CAD renderings across 12 classes, a real validation set of MS COCO images, and a test set of YouTube Bounding Boxes frames (VisDA). It allows a clean inductive check. We fit the refinement and the linear map on COCO validation features, then apply the map unchanged to the YouTube test split. If the gains survive this transfer, the method generalizes to new images rather than memorizing the target set it was fit on. We run it on two frozen backbones, a synthetic-trained ResNet and a frozen CLIP image encoder, to test whether repulsion matters more for the less invariant one.
The few-shot regime uses MNIST→SVHN under DIRL's 10-labels-per-class protocol. We will add DIRL's Pybullet→Toyota HSR data only if it can be obtained, since the file gives no release.
A stretch experiment applies the refinement to per-region features of a frozen segmentation encoder on GTA5→Cityscapes, scored in mIoU. The source-only reference there is VisDA-S's "21.4" mIoU and the ceiling is VLTSeg's fine-tuned 63.1% DG mean.
Three checks decide the project:
- The $\lambda_R$ ablation must show gains beyond attraction-only refinement on both VisDA-C and MNIST→SVHN.
- Neighbor purity must move in the predicted direction, and only when repulsion is on.
- A noise sweep that randomly corrupts a growing fraction of the constraint pairs must show graceful degradation. This matters because the unsupervised regime depends on agreement-filtered pseudo-pairs, and the research file names lexicon quality as retrofitting's main limitation.
All runs report accuracy, refinement wall-clock time, and peak memory, so the cost argument is checked as carefully as the accuracy argument.
References
- Counter-fitting Word Vectors to Linguistic Constraints
- DIRL: Domain-Invariant Representation Learning for Sim-to-Real Transfer
- Self-ensembling for visual domain adaptation
- Strong but Simple: A Baseline for Domain Generalized Dense Perception by CLIP-Based Transfer Learning
- Retrofitting Word Vectors to Semantic Lexicons
- More Discriminative Sentence Embeddings via Semantic Graph Smoothing
- VisDA: The Visual Domain Adaptation Challenge
- Model Fusion via Retrofitting