When a model trained on synthetic images labels real images for itself, its mistakes are not random. A classifier trained on CAD renders, with flat textures and studio lighting, tends to be confidently wrong in a systematic way on real photos. Retraining on those mistakes, a process called self-training, teaches it to repeat them. We propose to decide which of the model's own labels to trust using a feature summary that is blind, by construction, to the kind of change a simulator most often gets wrong: how strongly each feature channel responds. That summary is the matrix of correlations between feature channels inside each image. A rescaled channel leaves this matrix exactly unchanged, while the pattern of which object-part detectors fire together survives.
The bet is concrete. Under synthetic-to-real shift, rendering and lighting errors mostly change the magnitude of individual feature channels. If that holds, a pseudo-label reliability score computed from distances between correlation matrices should separate right labels from wrong ones much better than softmax confidence on the real target. Better selection of pseudo-labels should then translate into better adapted accuracy on VisDA-2017 and the smaller synthetic-to-real digit and sign benchmarks. The hypothesis can fail cleanly, and the proposal is built to find out whether it does.
Core Claim
Existing defenses against pseudo-label noise in synthetic-to-real adaptation score reliability in a space that still contains the nuisance the simulator introduced. Self-Ensembling for Visual Domain Adaptation won the VisDA-2017 challenge by accepting a teacher prediction only when softmax confidence exceeded 96.8%. Softmax is known to be overconfident under shift, a point AdaConG uses to motivate replacing it with conformal prediction sets. However, AdaConG's prediction sets need a labeled calibration set exchangeable with the deployment data, and that set does not exist on an unlabeled real target. RSDA scores reliability geometrically: it fits a mixture of a Gaussian and a uniform distribution to distances between features and class prototypes on the unit sphere. The sphere removes only one global magnitude per sample.
Our claim is that the right quotient goes further. Let $\Sigma \in \mathbb{R}^{d \times d}$ be the covariance of $d$ channel responses across the spatial positions of one image. Let $D_\Sigma$ be its diagonal, which holds the per-channel variances. The correlation matrix
$$C = D_\Sigma^{-1/2}\, \Sigma\, D_\Sigma^{-1/2}$$
is unchanged when $\Sigma \mapsto G \Sigma G$ for any positive diagonal $G$. A per-image, per-channel gain change therefore cannot move $C$. Riemannian Networks over Full-Rank Correlation Matrices (CorNets) argued in a non-adaptation setting that variances act as nuisance noise that hides relational structure. They reported correlation networks beating SPDNet by +17.69% on HDM05 and +13.84% on NTU120. We transfer that argument to the regime where the nuisance is the simulator itself. We also move it from the classifier to the pseudo-label filter, which is where errors compound.
A tension in the file needs to be faced directly. Every SPD-based domain adaptation method in it aligns covariances, not correlations: Log-D-CORAL, parallel transport on the SPD cone, and optimal transport on SPD matrices. If CorNets is right, those methods spend their alignment effort on exactly the component that is least reliable. Our experiments test which side is right in the synthetic-to-real case.
Research Program
The model is a standard ResNet-50 backbone, chosen to match the VisDA literature. On top of it sits a small second-order branch. A 1×1 convolution reduces an intermediate feature map to $d = 64$ channels at 14×14 resolution, giving 196 spatial samples per image. This makes the per-image covariance full rank up to a small ridge term, which avoids the rank collapse that breaks mini-batch covariance methods when the batch is much smaller than the channel count.
We form $C$ for each image and process it with an intrinsic correlation layer and the Riemannian logistic regression of CorNets. We use one of its zero-curvature metrics (OLM or LECM, which are cheap and have analytic gradients) and its curved Poly-Hyperbolic-Cholesky metric as a variant. The bilinear-map design of SPDNet serves as the covariance counterpart.
The pseudo-label loop builds on the Mean Teacher of French et al. and keeps its separate source and target batch normalization and its class-balance term. Only the reliability score changes. For each target image with teacher label $k$, we compute the geodesic distance $\delta$ from $C$ to the Fréchet mean of class $k$ on the correlation manifold. Before this, the target domain's mean correlation is recentered onto the source mean, as parallel transport recentering does for EEG (Yair et al.). We then fit RSDA's Gaussian-uniform mixture to these distances per class. The posterior
$$w(x) = \frac{\pi_k\, \mathcal{N}(\delta;\, 0, \sigma_k^2)}{\pi_k\, \mathcal{N}(\delta;\, 0, \sigma_k^2) + (1-\pi_k)\, u_k}$$
weights the consistency loss. Here $\pi_k$ is the inlier fraction, $\sigma_k$ the inlier spread, and $u_k$ the uniform outlier density for class $k$. We reuse the mixture machinery from RSDA. What is new is the space it is fitted in and the recentering step.
To prevent minority classes from receiving no pseudo-labels, a failure noted for thresholding by both French et al. and Generalize then Adapt, $\pi_k$ is estimated per class rather than globally. Checkpoints and the ridge and recentering hyperparameters are chosen without target labels using Pseudo-Label Validation. That method trains an undersmoothed imputer on held-out source data to build a surrogate target loss, and it recovered the oracle epoch on Camelyon17 (73% versus 53% for naive source validation). We also report oracle selection so that the two are not confused.
The diagnostic study is as important as the method. On a source-only model, we decompose the synthetic-to-real gap into its variance part and its correlation part. For each class, we measure how far source and target move apart in the diagonal of $\Sigma$ versus in $C$. Then we attribute each wrong confident teacher prediction on the real target to one of the two parts. A model fed only the variances, or only the correlations, tells us which part carries the error.
Baselines
The naive reference point is source-only training of the same ResNet-50 evaluated on the real target. The VisDA paper documents how severe this is, with AlexNet falling from 99% on synthetic to 28% on real COCO (VisDA). We rerun it on our backbone rather than quoting it.
The strongest existing competitor is the self-ensembling Mean Teacher with softmax thresholding of French et al. It is a pure pseudo-label noise method, it won the synthetic-to-real benchmark we target, and our method differs from it only in the reliability score. The file reports 92.8% test accuracy on VisDA-2017 for this method (Self-Ensembling for Visual Domain Adaptation). That was a challenge entry with its own engineering, so we also rerun it on our backbone, with our augmentations, training schedule, and the same number of hyperparameter trials selected by the same label-free validator. Beating it means a consistent gain in mean per-class target accuracy across three seeds on the rerun. We will also report honestly where we stand against the 92.8% figure.
The geometric competitor is RSDA. It uses the same mixture posterior on the sphere, so it isolates the effect of our invariance. The file gives RSDA no VisDA number, only 91.1% on Office-31, so we rerun it. DSGK's easy-to-hard spherical curriculum adds a second geometric reference, with 85.0% reported on VisDA-2017 (DSGK). AdaConG with a source-split calibration set stands for conformal reliability under a broken exchangeability assumption. For Syn-signs→GTSRB and Syn-digits→SVHN there is no reported result found on Lacuna for any of these methods, so all numbers there come from our reruns.
Three ablations isolate the key ingredient. The first swaps correlation for covariance in the same branch, testing whether removing variances matters. The second keeps correlation matrices but scores reliability with Frobenius distance, or with a single log-map to a tangent space and a Euclidean classifier, testing whether the intrinsic geometry matters beyond the invariance. The third removes the recentering step.
Why not just…?
Why not just keep softmax thresholding, which already won VisDA? It is a real threat on accuracy, because a high threshold plus class balancing is a strong recipe. Its known weakness is calibration on the target, which AdaConG documents for softmax under shift. The decisive measurement is round-zero reliability: the area under the ROC curve for separating correct from incorrect teacher labels on the real target, before any adaptation. If softmax confidence matches our score there, our method cannot win through better selection, and we will report that.
Why not just normalize away gain with batch or instance normalization? This is the most serious alternative. French et al. already use separate batch normalization per domain, which removes domain-level per-channel scale. Instance normalization would remove per-image scale in a Euclidean network. Neither gives a reliability score, but either might shrink the gain-driven errors enough that the score barely matters. We test this directly with three variants of the French baseline: with per-domain normalization, with per-image channel standardization, and with our correlation score added on top of both. If the score adds nothing once per-image standardization is present, the conclusion is that the invariance, not the manifold, did the work. The proposal would then reduce to a standardization result, and we would say so.
Why not just project correlations to a tangent space once and use ordinary layers? The work by Chen et al. on tangent-space flattening argues that it distorts distances far from the reference point. In domain shift, target points are by definition far from the source mean. Our recentering step partly undoes that argument, though, so the tangent-space ablation is a genuine test rather than a strawman. Because CorNets' zero-curvature metrics are already close to log-Euclidean, we expect the gap to be small. The curved metric is where intrinsic geometry could plausibly matter.
Why not just rely on a strong pretrained backbone? Pretrained real-image features hide much of the gap, as the VisDA authors warn. Yet the Pseudo-Label Validation results show DINOv2 fine-tuning still overfitting to the source domain. We report the main comparison with both ImageNet and DINOv2 initialization. A method whose gain disappears with the stronger backbone has limited value, and we will state it if that happens.
Proposed Experiments
The primary benchmark is VisDA-2017 classification: 152k synthetic CAD renders as source, real COCO validation images as target, evaluated by mean per-class accuracy. The secondary benchmarks are Syn-signs→GTSRB and Syn-digits→SVHN from French et al., where the generator is simpler and gain-type shifts can be controlled. HDM05 and Radar, the correlation benchmarks from CorNets, are used only to sanity-check our correlation layers against the reported margins over SPDNet.
Beyond target accuracy, we track three quantities throughout training. The first is the reliability ROC area. The second is the error rate among the pseudo-labels actually admitted to training. The third is per-class pseudo-label coverage, since class starvation is the known failure of filtering.
Three checks decide the question. First, a controlled intervention: on GTSRB, we apply random per-channel gain changes and, separately, synthetic changes to the correlation structure, by mixing channels with random rotations, to the backbone features of real images. Our score should be unmoved by the former and sensitive to the latter, and softmax confidence should not show that pattern. Second, the variance-versus-correlation attribution on VisDA must show that most confidently wrong teacher labels are driven by variance. If correlations carry most of the errors, the core hypothesis is false for CAD renders, and the method should not be expected to help. Third, the end-to-end comparison against the matched rerun of French et al. and against per-image standardization must show a gain that survives label-free checkpoint selection. A result that appears only under oracle selection does not count.
References
- Self-ensembling for visual domain adaptation
- Adaptive Conformal Guidance for Learning under Uncertainty
- Spherical Space Domain Adaptation With Robust Pseudo-Label Loss
- Riemannian Networks over Full-Rank Correlation Matrices
- Correlation Alignment by Riemannian Metric for Domain Adaptation
- Parallel Transport on the Cone Manifold of SPD Matrices for Domain Adaptation
- Optimal Transport on the Manifold of SPD Matrices for Domain Adaptation
- A Riemannian Network for SPD Matrix Learning
- Generalize then Adapt: Source-Free Domain Adaptive Semantic Segmentation
- Pseudo-Label Validation for Unsupervised Domain Adaptation
- VisDA: The Visual Domain Adaptation Challenge
- Deep Spherical Manifold Gaussian Kernel for Unsupervised Domain Adaptation
- Riemannian Multinomial Logistics Regression for SPD Neural Networks