Covariance Geometry Under Ill-Conditioned Synthetic Features

Models trained on rendered images often fail on real photos. A standard, cheap fix is to make the network's internal features have the same spread on synthetic and real data. Concretely, it penalizes the difference between the two feature covariance matrices, which record how feature dimensions vary together. This penalty is almost always the plain entrywise squared difference. Synthetic data has a specific pathology here. Renderings of CAD models (computer-aided-design meshes) vary much less than real photos, so their feature covariances are close to singular: many directions carry almost no variance. Mini-batch covariances are also rank-deficient whenever the batch is smaller than the feature dimension. Optimization theory on curved spaces says that the geometry used to compare such matrices sets how fast and how stably alignment can converge. How badly each geometry degrades depends on how ill-conditioned the matrices are.

This note proposes to test that prediction directly in the sim-to-real setting. The flat squared-difference penalty ignores the curved space where covariance matrices live. The classical curved geometry for covariances, the affine-invariant metric, becomes undefined as matrices approach singularity. The Bures-Wasserstein geometry, equal to the optimal-transport distance between Gaussians, stays finite for singular matrices and has non-negative curvature. We propose to make the conditioning of synthetic covariances an experimental knob. We then measure how alignment speed, gradient stability and final real-image accuracy change under each geometry as that knob is turned. We also check whether any benefit survives on top of the pseudo-labeling pipelines that actually win sim-to-real benchmarks.

Core Claim

Our claim is that in synthetic-to-real transfer, the choice of covariance geometry matters in proportion to how ill-conditioned the synthetic covariances are. Bures-Wasserstein alignment should degrade gracefully as conditioning worsens. Affine-invariant and log-Euclidean alignment should need growing regularization and eventually lose the alignment signal. The flat Frobenius penalty should stay stable but move slowly along directions with little variance. The novelty is not a new loss. It is a falsifiable, conditioning-dependent prediction imported from Riemannian optimization theory, and a controlled test of that prediction on a large sim-to-real benchmark where no one has run it.

Two lines of optimization work motivate the claim. First-order Methods for Geodesically Convex Optimization shows that on curved spaces, sectional curvature and domain diameter enter iteration complexity as a multiplicative penalty. Sectional curvature measures whether shortest paths spread apart or bend together. On Riemannian Optimization over Positive Definite Matrices with the Bures-Wasserstein Geometry shows that Bures-Wasserstein has non-negative curvature and depends linearly rather than quadratically on the matrix. It reports convergence "orders of magnitude faster" than affine-invariant or log-Euclidean methods as condition numbers grow. That paper tests only generic problems (Lyapunov equations, metric learning) and explicitly leaves domain adaptation open. The same paper also constrains us: it finds affine-invariant superior for log-determinant objectives. Our prediction is therefore not "Bures-Wasserstein always wins" but "it wins where conditioning is bad."

We must be candid about one technical wrinkle. The Bures-Wasserstein distance between covariances $A$ and $B$ is

$$d_{BW}^2(A,B)=\mathrm{tr}\,A+\mathrm{tr}\,B-2\,\mathrm{tr}\big(A^{1/2}BA^{1/2}\big)^{1/2}.$$

It is finite for singular matrices. The affine-invariant distance, $\|\log(A^{-1/2}BA^{-1/2})\|_F$ with $\|\cdot\|_F$ the Frobenius norm, is not. However, the gradient of $d_{BW}^2$ with respect to $A$ involves the optimal-transport map, which also grows as eigenvalues of $A$ approach zero, because the matrix square root is not smooth at zero. Theory therefore does not settle how gradient sizes compare under each geometry during end-to-end training. That is exactly why the question needs measurement rather than assertion.

Research Program

The program has two layers.

The first layer is a controlled study of the alignment subproblem. Here the curvature-dependent convergence theory applies literally, because the variable being optimized is itself a covariance matrix. Parallel Transport on the Cone Manifold of SPD Matrices for Domain Adaptation aligns EEG covariances from several subjects by transporting each to an affine-invariant "mean of means." We reuse that construction for a different purpose. Our domains are several synthetic rendering styles plus the real domain, and the shared target is their barycenter, meaning the covariance that minimizes the summed squared distance to all of them. We compute this barycenter by Riemannian gradient descent under affine-invariant and under Bures-Wasserstein geometry. For more than two domains neither geometry gives a closed form, so iteration counts are meaningful. We record iterations to a fixed tolerance as a function of the condition numbers of the input covariances. This mirrors the matrix Karcher-mean benchmark of First-order Methods for Geodesically Convex Optimization, but uses real feature covariances extracted from synthetic and real images.

The second layer is end-to-end training. We start from the Deep CORAL recipe of Deep CORAL: Correlation Alignment for Deep Domain Adaptation, which adds a Frobenius covariance penalty to the source cross-entropy. We then swap the penalty among four options: Frobenius, log-Euclidean, affine-invariant, and Bures-Wasserstein. The last three are computed on a 256-dimensional bottleneck by eigendecomposition, at cubic cost in that dimension. For each run we log per-step gradient norms of the alignment term, the fraction of steps with gradient spikes, and steps to reach a fixed real-validation accuracy.

Conditioning is manipulated through three knobs. The first is the ratio of mini-batch size to feature dimension; covariances are exactly singular below one. The second is the amount of shrinkage toward the identity. The third is synthetic diversity, controlled by training on subsets that keep only a few feature-space clusters of renders per class. This mimics weaker domain randomization, meaning less random variation of lighting, pose and texture in the renderer.

We do not pursue the hyperbolic direction of When Cars Meet Drones. The barrier proved in Negative curvature obstructs acceleration concerns the impossibility of acceleration, not conditioning-driven instability, so it would be a separate study.

Baselines

The naive reference point is source-only training with no adaptation. On VisDA-C, VisDA: The Visual Domain Adaptation Challenge reports that AlexNet "drops from 99% accuracy on synthetic data to just 28% on real COCO images." We rerun source-only with our backbone so that every comparison uses the same network.

The direct competitor is Deep CORAL with the Frobenius penalty. It is the method whose geometry we question, and on Office it reaches 72.1% average accuracy in Deep CORAL. We found no reported result on Lacuna for Deep CORAL on VisDA-C, so we run it ourselves. The distribution-matching reference is DAN, which aligns feature means under Maximum Mean Discrepancy (a kernel-based distance between distributions) and reaches 51.62% on VisDA-C per the VisDA paper.

The closest existing method is the Conditional Kernel Bures loss of Conditional Bures Metric for Domain Adaptation. It already uses a Bures-type distance, applied to class-conditional kernel covariance operators with pseudo-labels, and reports 68.7% on Office-Home and 90.2% on ImageCLEF-DA. We found no reported result on Lacuna for it on VisDA-C. It never varies conditioning or measures convergence. Because it is the strongest competitor on the specific question of whether Bures geometry helps, we run it with its conditional pseudo-labeled form.

The strongest system overall is the pseudo-labeling pipeline that won VisDA-C at 92.8%. We reimplement a pseudo-labeling baseline on our backbone and add each covariance penalty as a regularizer on top of it.

The ablation that isolates our key ingredient is affine-invariant and log-Euclidean alignment with tuned shrinkage. It separates "curved geometry" from "geometry that tolerates singular matrices."

Fairness rules apply to all methods. They share the backbone, data, optimizer, and number of training steps. Each gets the same hyperparameter budget of twenty trials over alignment weight, learning rate and, where applicable, shrinkage, selected on the COCO validation split. Results are reported on the YouTube test split over three seeds.

We define beating a baseline in two parts. First, Bures-Wasserstein must reach the Frobenius baseline's final accuracy in fewer steps and with fewer gradient spikes. Its advantage must also widen monotonically as conditioning worsens. Second, on top of pseudo-labeling it must add accuracy beyond seed variance. A flat advantage that does not track conditioning would count against the claim even if Bures-Wasserstein wins on average.

Why not just…?

Why not just add a ridge term to the covariances and use the affine-invariant metric? This is the most serious threat. Shrinkage toward the identity makes every covariance invertible, and it may remove most of the difference between geometries. Our prediction is that the ridge needed for stability grows with ill-conditioning, and that such a ridge floods the low-variance directions, which is exactly where synthetic and real features differ. The shrinkage sweep settles it. If affine-invariant alignment with a modest ridge matches Bures-Wasserstein across all conditioning levels, the claim is wrong and we will report that.

Why not just use a closed-form whitening and re-coloring map, like linear CORAL, with no iterative optimization? For two domains a closed form exists, and Deep CORAL reports 70.4% for linear CORAL on Office, only slightly below the deep version. Closed forms have two limits. They do not extend to barycenters of many rendering styles, where iterative Riemannian optimization is required. They also fix features once instead of shaping them during training. We include linear CORAL in the first layer as a two-domain reference.

Why not just skip covariance alignment and use pseudo-labeling, since it wins by forty points? This is also a real threat. The VisDA gap between DAN (51.62%) and pseudo-labeling (92.8%) suggests covariance geometry may be a second-order effect in absolute accuracy terms. We therefore test the regularizer on top of pseudo-labeling rather than instead of it. There, stability under small batches may matter most, because noisy pseudo-labels already stress training. If the gain vanishes there, the contribution shrinks to a convergence-and-stability finding, and we will say so.

Proposed Experiments

The main benchmark is VisDA-C: 152,000 synthetic CAD renders transferred to COCO and YouTube frames. A secondary check uses the CAD-to-PASCAL VOC 2007 setting. There Synthetic to Real Adaptation with Deep Generative Correlation Alignment Networks reports 18.48% for raw renders, 23.97% for DAN, and 27.46% for its pixel-level Frobenius alignment. We would replace its Frobenius naturalness loss with Bures-Wasserstein and count the pixel iterations needed to reach its accuracy, since the authors themselves flag iteration cost.

The decisive checks are three. The first is whether measured source covariance condition numbers on VisDA-C are in fact far larger than real ones; if not, the premise fails early and cheaply. The second is whether the log-log slope of barycenter iterations against condition number is smaller under Bures-Wasserstein than under affine-invariant geometry. The third is whether the end-to-end accuracy and stability gap between Bures-Wasserstein and the best-tuned alternative grows as batch-to-dimension ratio and synthetic diversity shrink.