Robust Subspace Cleaning of Pseudo-Labels for Synthetic-to-Real Adaptation

Models trained on synthetic images and then adapted to real ones usually adapt by self-training. The model labels unlabeled real images itself, keeps the labels it is confident about, and retrains on them. The weak point is that confident labels are often wrong under a large domain gap, and every wrong label pulls the model further toward its own mistakes. This note proposes a different test for which pseudo-labels to trust. Group the real images by their predicted class. Ask whether each image's feature vector fits the low-dimensional subspace that the class's correctly labeled images share. Images that do not fit are likely mislabeled.

The fitting step has to be robust. If the subspace is estimated from a contaminated group, the mislabeled images drag the subspace toward themselves and hide. It also has to avoid a known bias of classical robust low-rank methods: they shrink the dominant directions of the data along with the noise. Selective singular value penalization fixes that bias. It penalizes only the tail of the singular value spectrum and leaves the leading directions alone. We argue this matters specifically for synthetic-to-real transfer, because the directions in which real images differ from synthetic ones are moderately strong. Uniform shrinkage would weaken those directions and wrongly flag correct but "very real-looking" images as noise.

Core Claim

Here is the method in brief. For each predicted class $c$ in a self-training round, stack the target-domain features of the images assigned to $c$ as columns of a matrix $F_c \in \mathbb{R}^{d \times n_c}$, where $d$ is the feature dimension and $n_c$ the number of such images. Then solve

$$\min_{A_c, E_c} \sum_{i > r_c} \sigma_i(A_c) + \lambda \|E_c\|_{2,1} \quad \text{s.t.} \quad F_c = A_c + E_c,$$

The terms are as follows. $\sigma_i(A_c)$ are the singular values of the clean low-rank part $A_c$, sorted in decreasing order. $r_c$ is the number of leading singular values left unpenalized. $\|E_c\|_{2,1}$ is the sum of the Euclidean norms of the columns of $E_c$, which pushes whole columns (whole images) to be either clean or outliers. $\lambda$ trades off the two terms. The column norm $\|E_c^{(j)}\|_2$ then scores how likely image $j$'s pseudo-label is wrong. Flagged images are dropped or downweighted before retraining.

The proposal makes three specific claims, each of which can be tested:

  1. This column-outlier score detects wrong pseudo-labels better than confidence alone.
  2. Leaving the top $r_c$ directions unpenalized detects them better than a uniform nuclear-norm version.
  3. The free quantities $r_c$ and $\lambda$ can be set without any real labels. The synthetic source domain has clean labels, so it supports estimating class ranks and simulating label noise for calibration.

The third point turns the domain gap from a nuisance into a resource. It also addresses the main open problem the file identifies for selective penalization: choosing the rank or cap is brittle when no ground-truth labels are available.

Relation to Prior Work

The motivating evidence comes from the pseudo-label literature, and three papers report three stages of the same failure.

  1. Learning Transferrable Representations for Unsupervised Domain Adaptation learns transferable representations jointly with target labels. It observes that early pseudo-labels are especially noisy and that unfiltered early errors distort the learned features.
  2. Tri-net for Semi-Supervised Deep Learning uses three networks that label data for each other. It shows that unverified pseudo-labels accumulate errors across self-training cycles.
  3. Self-Ensembling Attention Networks: Addressing Domain Shift for Semantic Segmentation applies self-ensembling to segmentation under domain shift. It reports that target cluster representations drift away from semantic boundaries when initial predictions are corrupted.

All three detect or limit noise through agreement between models or through confidence. None asks whether a labeled image is geometrically consistent with the rest of its class. That geometric question is what we add.

The method itself comes from low-rank recovery.

These low-rank works all recover a signal from measurements with known semantics. None uses the decomposition to audit labels a model assigned to itself, and none studies two domains with different spectra.

A caveat on novelty: the research file I worked from was cut off before its section on closest existing work. Spectral regularizers on batch features or predictions are, to my knowledge, an existing line in domain adaptation, but the file contains none of them, so I cannot cite them or check their numbers. The distinction I can defend is in the target of the method. Those regularizers, as I understand them, reshape the network's training loss. This proposal instead uses a per-class robust decomposition as a pseudo-label filter whose decisions can be measured directly.

Research Program

Build. The system is a standard self-training loop for synthetic-to-real classification. In each round it does three things:

  1. It extracts penultimate-layer features for all target images.
  2. It projects them to 256 dimensions with a projection fit on source features.
  3. For each predicted class, it runs the decomposition above using randomized partial SVDs.

The resulting outlier scores replace or combine with confidence when choosing which pseudo-labels to train on.

Analyze. The analysis has two parts.

The first concerns the spectra themselves. We compare per-class singular value spectra of source features, correctly labeled target features, and mislabeled target features (using held-out target labels only for analysis). The hypothesis behind the "selective" choice is concrete: correctly labeled real images span the source class subspace plus a few extra real-specific directions of intermediate strength. Mislabeled images instead load onto directions belonging to other classes.

The second concerns calibration. We estimate $r_c$ from clean source features as the smallest rank that captures a fixed fraction of class energy. We select $\lambda$ by corrupting source labels in a structured way, relabeling images as the class the model most often confuses them with, and maximizing detection AUC on those simulated errors.

The key risk is that ranks estimated on synthetic data are too small for real data. If the source rank is too small, real-specific directions fall into the penalized tail. We measure this directly, and the capped-norm variant is the fallback.

Baselines

Naive reference. The do-nothing reference is source-only training with no adaptation. The naive adaptive reference is confidence-thresholded self-training, which keeps every pseudo-label whose maximum softmax probability exceeds a fixed threshold, retrains, and repeats. This is the mechanism whose failure the three motivating papers document.

Fair comparison. All methods share the same backbone, source-pretrained checkpoint, number of rounds, and per-round retraining budget. Each gets the same number of hyperparameter trials, and all selection is done on simulated source noise, never on target labels. That last rule matters because a target-label-tuned threshold would overstate the baselines.

Strongest existing competitor. Among methods in the file, this is co-training-style agreement filtering in the spirit of Tri-net: three heads trained on different bootstraps, keeping a pseudo-label only when two agree. It is the closest existing answer to "which pseudo-labels are wrong," and it costs more training compute. We give it the same total number of training updates as our method plus the ensemble overhead. I found no reported result on Lacuna for Tri-net or confidence-thresholded self-training on synthetic-to-real benchmarks. All numbers will therefore come from our own reruns.

Ablations isolating the key ingredient. The ablations are essential, since they decide whether selectivity and robustness each matter:

  1. Uniform nuclear-norm outlier pursuit, which is the same column-sparse decomposition but penalizes all singular values.
  2. Plain truncated PCA residual scoring, which fits the top-$r_c$ subspace by ordinary SVD on the contaminated group and scores images by reconstruction error.
  3. The capped-norm variant in place of the hard rank cutoff.
  4. Hard rank $r_c$ set from target-label oracle ranks, giving an upper bound on calibration quality.

What counts as beating a baseline. Our method beats a baseline if two things hold. First, mislabel-detection AUC is higher at matched retained-set size, averaged over rounds. Second, final target accuracy is higher across three seeds, with the gap exceeding seed standard deviation.

Why not just…?

Why not just raise the confidence threshold? A higher threshold keeps fewer but cleaner labels. Under a large domain gap, though, the confident errors are exactly the systematic confusions the motivating papers describe, such as synthetic textures mapping real objects to a wrong class with high confidence, and thresholding cannot remove those. The test is in the experiments: we plot detection AUC of confidence versus our score separately for high-confidence pseudo-labels. If confidence matches us there, the proposal has no reason to exist.

Why not just fit an ordinary PCA subspace per class and score residuals? This is the most serious threat, because it is far simpler. It works whenever mislabeled images are a small fraction of the group. Its weakness is that contamination tilts the fitted subspace, and the error grows with the noise rate, which is highest in early rounds, the stage Learning Transferrable Representations for Unsupervised Domain Adaptation identifies as decisive. We will stratify classes by their true pseudo-label noise rate. If truncated PCA matches the robust version even in the noisiest classes, we will report that and recommend it.

Why not just use uniform nuclear-norm robust PCA, which is convex and has no rank to choose? The photometric-stereo gap reported by Partial Sum Minimization of Singular Values in RPCA for Low-Level Vision shows how much uniform shrinkage can cost when signal directions are strong. Our specific prediction is narrower: uniform shrinkage will falsely flag correctly labeled images that lie far from the source subspace. We test this by measuring false-flag rates as a function of each image's residual from the source class subspace. If the uniform version's false flags are not concentrated among the most "real" images, the selective term is not earning its non-convexity.

Why not just compare features to class prototypes or use small-loss selection? Prototype distance treats each class as a single point. It cannot distinguish a real image that varies along a legitimate within-class direction from one drifting toward another class. We include prototype distance as an extra baseline under the same budget, and expect it to lose most on classes with a high-rank source spectrum.

Proposed Experiments

Settings. The main setting is synthetic-to-real image classification in the VisDA style (rendered objects to real photos). A secondary setting applies the same filter to per-class pixel features in GTA-to-Cityscapes-style segmentation, the regime studied by Self-Ensembling Attention Networks: Addressing Domain Shift for Semantic Segmentation. The file gives no reported numbers for either benchmark. Target labels are used only for evaluation and analysis.

Metrics. Because the target labels exist, we can measure the filter itself, not just final accuracy: per-round precision, recall, and AUC for detecting wrong pseudo-labels. We also report final target accuracy (or mean IoU for segmentation) and wall-clock overhead per round.

Decisive checks. Four results settle the proposal:

  1. The robust selective score must beat confidence on high-confidence errors.
  2. It must beat truncated PCA in the noisiest early rounds.
  3. It must beat uniform robust PCA specifically on real-looking correct images.
  4. Ranks calibrated on synthetic data must come within a small margin of oracle target ranks.

If the third check fails, the selective penalty adds nothing over robust PCA. If the fourth fails while the capped variant succeeds, the conclusion is that a soft cap, not a rank, is what carries across the domain gap. Either result is publishable and clarifies when spectral structure can audit self-generated labels.