Two Wrong Teachers Make a Right Label: Sim-to-Real Adaptation as Multi-Annotator Noise Estimation

When a classifier trained on rendered 3D objects is applied to real photographs, its mistakes are not random. A model trained on clean synthetic cars and trucks keeps calling real trucks "cars," and real knives "skateboards," because the simulator got some shapes and textures wrong in consistent ways. A vision-language model trained on web image-caption pairs, such as CLIP, also makes consistent mistakes on the same photographs. Its mistakes have a different cause, though: captions that mention co-occurring objects or that describe the page instead of the picture. This proposal treats these two models as two noisy annotators whose errors are structured but have different origins. It adds a third annotator that sees no labels at all: clusters of the real images found by self-supervised learning. From the pattern of agreement among the three, it recovers each annotator's confusion pattern on the real domain without any real labels. It then trains the target model on the resulting corrected labels.

The core claim is that sim-to-real pseudo-label noise is asymmetric label noise, meaning noise where the probability of flipping from class $i$ to class $j$ depends on $i$ and $j$. Because it is asymmetric, it can be estimated rather than merely dampened, provided there are enough independent views. Current source-free adaptation methods treat target pseudo-label noise as a generic nuisance and regularize against memorizing it. The proposal instead borrows the identifiability argument from the label-noise literature and supplies the independent views it requires from the web-trained model and from target-only clustering.

What exists and what is missing

Three findings motivate the idea. When Source-Free Domain Adaptation Meets Learning with Noisy Labels studies adaptation without access to source data. It shows that pseudo-labels produced by a synthetic-trained model on VisDA-C carry "unbounded" noise, with per-sample mislabeling probability approaching 1. Standard robust losses such as generalized cross-entropy give only marginal gains in that setting. The paper's remedy, early-learning regularization (ELR), uses a running average of predictions to slow memorization. ELR is class-agnostic: it never asks which classes are being confused. Identifiability of Label Noise Transition Matrix supplies the method we reuse. Using Kruskal's theorem, it shows that a noise transition matrix is identifiable from three conditionally independent views of the true label, and that class-disentangled representations can act as such views. It demonstrates this only for synthetic noise on CIFAR. Training Convolutional Networks with Noisy Labels constrains the design. Its learned linear noise layer absorbs class-conditional flips, but without extra views such a layer is under-determined and can settle on the wrong matrix.

On the web side, Align before Fuse and BLIP show that web-trained vision-language models absorb noisy captions in specific ways. ALBEF uses soft momentum-teacher targets, and BLIP generates and then filters captions. Neither treats the resulting model as an annotator with a measurable, class-structured error pattern. Multi-Objective Interpolation Training uses neighborhood disagreement in a learned feature space to detect asymmetric flips, on both CIFAR-100 and mini-WebVision. It never crosses a rendering domain gap.

No work in this set combines a simulator-trained model and a web-trained model as complementary noisy annotators on a sim-to-real target, and none estimates their confusion matrices there.

Core Claim

We call the views $v_1$ (the synthetic-trained classifier's prediction), $v_2$ (the web-trained model's zero-shot prediction from class-name prompts), and $v_3$ (the cluster index assigned by k-means on features from a self-supervised encoder trained only on unlabeled real target images). The true but unknown class is $y$. If the three views are approximately independent given $y$, the joint distribution of their outputs on the target set factorizes as

$$ P(v_1, v_2, v_3) = \sum_{y=1}^{K} \pi_y \, T^{(1)}_{y v_1} \, T^{(2)}_{y v_2} \, T^{(3)}_{y v_3}, $$

where $K$ is the number of classes, $\pi_y$ is the target class prior, and $T^{(k)}_{y v}$ is the probability that view $k$ outputs $v$ when the truth is $y$. The cluster view may use more than $K$ clusters, since Kruskal's conditions allow views with different output alphabets. Under the rank conditions in Identifiability of Label Noise Transition Matrix, all three matrices and $\pi$ are recoverable up to a relabeling of $y$. The relabeling is pinned down by assuming each classifier view is correct more often than not on its diagonal.

We fit this model with expectation-maximization (EM) initialized by spectral tensor decomposition. This is the same model as Dawid–Skene crowdsourcing, with machines as the crowd. The fit yields a per-image posterior $q(y \mid v_1, v_2, v_3)$. The target model is then trained on these soft posteriors, with ELR applied on top so that residual errors are not memorized.

The specific prediction is that the gain will concentrate on class pairs where one view's errors are asymmetric and the other view is reliable. Examples are car versus truck, where the web model is likely stronger, and rare object types whose web captions are sparse, where the simulator view is likely stronger. Simple averaging of the two models' probabilities cannot exploit this kind of per-class, per-direction reliability.

Research Program

The work has three parts.

The first is measurement. On VisDA-C validation, ground-truth labels exist, so we can compute each view's true confusion matrix on real images. We can also directly test the conditional-independence assumption by measuring how often the simulator and web views make the same wrong prediction, compared with what independence predicts. This test is decisive. The paper "Contrastive Learning Improves Model Robustness Under Label Noise" (listed with the other baselines below) warns that semantically close classes share features. If both views fail on car versus truck for the same visual reason, their errors correlate and identifiability weakens. In that case we extend the model with a per-class-pair correlation term fit on the cluster view, and we report how much accuracy the correlation costs.

The second is estimation. We fit the three-view model on unlabeled target images and report how far the estimated transition matrices are from the true ones, using the mean absolute error metric of the identifiability paper. On CIFAR-10 at 30% noise, that paper reports 4.42 with SimCLR features and 3.73 with disentangled features. Those numbers come from a different setting and serve only as rough scale anchors.

The third is adaptation. We train the target model on the posteriors and compare it against the baselines below, with every method given identical access to the web-trained model.

Baselines

The naive reference point is the source-only synthetic model evaluated on real images. VisDA reports that AlexNet falls from 99% on synthetic data to 28% on real COCO images (VisDA). We will rerun source-only with our backbone so that every later gain is measured from the same starting point.

The second reference is the web-trained model's zero-shot accuracy alone. VisDA itself warns that strong pre-training can inflate apparent adaptation gains. No reported result was found on Lacuna for CLIP zero-shot on VisDA-C, so we will measure it ourselves. Any claimed gain must clear it.

The strongest existing source-free method in the file is G-SFDA with ELR. It reports 86.4% accuracy on VisDA-C, against 84.8% for G-SFDA alone (When Source-Free Domain Adaptation Meets Learning with Noisy Labels). On Office-Home and DomainNet, the same paper reports NRC with ELR at 72.6% and 79.4%. To make the comparison fair, we will run G-SFDA+ELR and NRC+ELR in two forms: as published, and with the web model's prediction added as an extra pseudo-label source by probability averaging. Both forms use the same backbone, the same target images, and the same hyperparameter search budget. Beating the published 86.4% alone would not be enough. The proposal must beat the web-augmented G-SFDA+ELR by a margin larger than seed variance over at least three seeds. The 92.8% VisDA challenge winner used source data and a mean teacher, so we report it as context rather than as a source-free competitor.

Two ablations isolate the key ingredient, which is estimating asymmetric per-view confusion. The first is two-view Dawid–Skene without the cluster view. Theory says this setting is not identifiable, so it should be unstable or worse. The second is the three-view model with each $T^{(k)}$ constrained to be symmetric, meaning a single uniform flip rate per view. If the symmetric version matches the full model, the asymmetry claim fails. A third ablation adds a learned linear noise layer in the style of Training Convolutional Networks with Noisy Labels on top of averaged pseudo-labels. This tests whether end-to-end noise absorption, without the extra views, recovers the same matrices.

Why not just…?

Why not just average the synthetic model and CLIP, then self-train? This is the real threat, and the web-augmented baseline above exists to settle it. Averaging gives each model one global weight. If the simulator view is confidently wrong on trucks but reliable on aeroplanes, averaging still inherits its truck errors whenever its confidence exceeds the web model's. The prediction is that our gain over averaging comes almost entirely from the classes where the true confusion matrices measured on validation are most off-diagonal and least aligned between the two views. If the gain is uniform, or absent, the per-class modeling is not doing the work, and we will say so.

Why not just fine-tune CLIP on the synthetic data and use one model? Fine-tuning on renders pulls the web model toward simulator shortcuts and makes its errors correlated with the synthetic view. That correlation is exactly what destroys the independence the estimator needs. We will run this baseline directly. We will also measure error correlation before and after fine-tuning, which shows whether the single merged model loses the complementarity.

Why not just apply a robust loss or confidence thresholding to the pseudo-labels? The source-free noisy-label paper already reports only marginal gains from robust losses on VisDA-C, because the noise is unbounded. Systematic confusions also tend to be high-confidence, so a confidence threshold keeps the wrong labels it most needs to drop (Multi-Objective Interpolation Training). We include confidence-thresholded self-training at 0.95 as a cheap baseline for completeness.

Why not just spend a small labeled target set to estimate the confusion matrices? That is a sensible engineering option, and we include it as an oracle upper bound: matrices estimated from 1% of labeled validation images. If the unsupervised three-view estimate gets within a point of this oracle, the method removes the need for target labels. If it does not, the oracle quantifies what a little labeling buys.

Proposed Experiments and Decisive Checks

The main benchmark is VisDA-C, with synthetic CAD renders as source and real COCO validation and YouTube test frames as target, across 12 classes. The YouTube frames are themselves web-sourced, which keeps the web-noise regime realistic. Office-Home and DomainNet test whether the method holds beyond rendering gaps; there the gaps are stylistic, and we expect smaller but nonzero gains.

Metrics are per-class and mean accuracy, transition-matrix estimation error against the true validation confusions, and observed versus independence-predicted joint error rates between views.

Three outcomes would falsify the proposal. The first is cross-view error correlation on VisDA-C validation so high that three-view estimation error is no better than two-view. The second is the symmetric-matrix ablation matching the full model. The third is web-augmented G-SFDA+ELR matching ours.

A positive result needs two things. Estimated matrices must track the true per-class asymmetries. Accuracy must improve over the web-augmented 86.4%-class baseline, with the improvement concentrated on the classes where the two teachers disagree in structured ways.

A secondary controlled check uses the same pipeline with the real target swapped for CIFAR-100 under 40% asymmetric noise, where MOIT+ reports 74.04%. The noise there is known exactly, so this verifies that the estimator recovers injected asymmetric matrices before we trust it on the sim-to-real target.