Collapse on Synthetic, Cluster on Real: A Geometric Account of the Sim-to-Real Gap

Models trained on rendered images often fail on photographs. A classifier trained on CAD renderings in the VisDA challenge drops from 99% accuracy on synthetic data to 28% on real COCO images. The usual explanation is "shortcuts": the network latches onto clean textures and canonical lighting that do not exist in the real world. We propose a more measurable version of that explanation. Synthetic data has little variation within each class, so supervised training on it pushes each class's features toward a single point. This endpoint is called neural collapse: the spread of features within a class shrinks toward zero, and the class centers settle into a fixed, maximally separated arrangement. Collapse throws away the within-class variation the real domain will need. Our claim is that sim-to-real transfer works best when features stay deliberately uncollapsed on the synthetic source and become tightly clustered on the unlabeled real target. Both quantities can be measured without real labels.

This matters for two reasons. First, it reconciles two findings in the transfer literature that look contradictory. Second, it gives practitioners something sim-to-real currently lacks: a label-free signal for choosing checkpoints and hyperparameters when no annotated real data exists. It also gives a cheap test of a question the hyperbolic-geometry literature has not asked. Are the reported gains of curved embedding spaces under domain shift really about curvature, or mostly about how much those spaces let classes collapse?

Core Claim

Let $\Sigma_W$ be the average within-class feature covariance, $\Sigma_B$ the covariance of class means, and $K$ the number of classes. The standard collapse statistic is

$$\mathrm{NC}_1 = \tfrac{1}{K}\,\mathrm{tr}\!\left(\Sigma_W \Sigma_B^{\dagger}\right),$$

where $\dagger$ is the pseudo-inverse. Small $\mathrm{NC}_1$ means tight, collapsed classes. We hypothesize that real-target accuracy rises with $\mathrm{NC}_1$ measured on held-out synthetic data, up to a point, and falls with $\mathrm{NC}_1$ measured on real features grouped by the model's own pseudo-labels. The method that follows from this has two stages. Synthetic training keeps the feature extractor uncollapsed while a separate head handles classification. Unsupervised adaptation on real data then drives target clusters to collapse.

The two papers this reconciles disagree only on the surface. Controlling Neural Collapse Enhances Out-of-Distribution Detection and Transfer Learning shows that strong collapse hurts downstream transfer. On VGG17, its uncollapsed encoder had 41.85% generalization error against 66.36% for the collapsed projector. It fixes this by training the feature extractor to maximize feature entropy, estimated from nearest-neighbor distances, while a frozen projector built from a simplex of equally separated class directions absorbs the collapse. Understanding and Improving Transfer Learning of Deep Models via Neural Collapse shows that transfer accuracy correlates with stronger collapse on the downstream task, and that over-collapsing on source data makes models brittle. Read together, the two papers say collapse is harmful on the source and helpful on the target. The sim-to-real setting is the most extreme test of that reading, because synthetic classes are unusually homogeneous. Neither paper studies domain adaptation. Both supply method and motivation rather than the claim itself.

Research Program

The first component is a measurement study. On VisDA-C and on the small synthetic-to-real pairs Syn-digits→SVHN and Syn-signs→GTSRB, we train source models across a deliberate grid of collapse levels. The grid varies training length, the weight of the entropy regularizer, label smoothing, and augmentation strength. For every checkpoint we record five quantities: source $\mathrm{NC}_1$ on held-out renderings, target $\mathrm{NC}_1$ on unlabeled real images under teacher pseudo-labels, local intrinsic dimension of real features, real-feature norm statistics, and true real accuracy. Local intrinsic dimension is included because Local Intrinsic Dimension of Representations Predicts Alignment and Generalization in AI Models and Human Brain reports that it tracks generalization, so it is a competing predictor. The test is whether the two-sided collapse signal ranks checkpoints by real accuracy better than these alternatives. The relevant measure is Spearman correlation, together with regret of the selected checkpoint against the best one.

The second component is the method. During synthetic training, the backbone receives the nearest-neighbor entropy term from Harun et al., and a frozen simplex head carries the classification loss. Adaptation then runs the self-ensembling recipe of Self-ensembling for visual domain adaptation on the backbone. That recipe is a teacher network that is a running average of the student, separate normalization statistics per domain, and confidence-thresholded pseudo-labels. We add one change: a term that tightens real-domain clusters around teacher pseudo-label centers. We expect gains to come mainly through better early pseudo-labels, since an uncollapsed source backbone should place real images of one class nearer each other.

The third component is the geometry question. HALO, from Hyperbolic Active Learning for Semantic Segmentation under Domain Shift, places pixel features in the Poincaré ball, a model of hyperbolic space in which volume grows exponentially with radius. It found that the distance from the center tracks how scarce a class is ($\rho=-0.899$). HyperFLAW, from When Cars Meet Drones, uses hyperbolic class prototypes across car and drone viewpoints and reports 28.96% mIoU against 26.96% for LADD. Neither paper measured within-class variance. We swap the classifier head among Euclidean, unit-sphere, and Lorentz geometries. The Lorentz model is the numerically stable hyperbolic form favored by Non-Euclidean Geometries for Prototype-Based Image Classification: a Reality Check. We compare the three heads at matched source $\mathrm{NC}_1$. If the hyperbolic advantage disappears once collapse is matched, curvature was mainly a collapse regulator in these settings. If the advantage persists, curvature contributes something beyond collapse.

Baselines

The naive reference is a source-only model trained on synthetic data with cross-entropy and no adaptation. We report it at our backbone and alongside the published AlexNet figure of 28% on real COCO from VisDA. The classic alignment reference is the Deep Adaptation Network, which matches source and target feature distributions with a kernel distance; VisDA reports 51.62%. The geometric alignment competitor is DSGK, from Deep Spherical Manifold Gaussian Kernel for Unsupervised Domain Adaptation. It aligns distributions along geodesics of the unit sphere and reports 85.0% on VisDA-2017.

The strongest competitor is Euclidean self-ensembling. It won VisDA-2017 with 92.8% test accuracy, per both VisDA and Self-ensembling for visual domain adaptation. It is the hardest bar because our method reuses its adaptation stage, so any gain must come from the source geometry. The 92.8% figure was obtained with the authors' own backbone and augmentation, so we will re-run self-ensembling ourselves on the same backbone and with an identical augmentation pipeline. Both systems get the same number of tuning trials and GPU-hours. Beating it means a gain on the re-run version that is consistent across three seeds and exceeds the seed spread, plus no loss on the minority classes.

Three ablations isolate the key ingredient. The first removes the entropy term but keeps the frozen simplex head. The second keeps the entropy term but adds no target-cluster term. The third is a "collapse-matched" control: plain early stopping chosen to reach the same source $\mathrm{NC}_1$ as our regularizer. This tests whether how collapse is limited matters, or only how much. For segmentation on GTA5→Cityscapes, the VisDA-S references are 21.4 mIoU source-only and 47.5 mIoU for the challenge winner. We found no reported unsupervised hyperbolic result on Lacuna for this pair. HALO's 74.5% uses 5% target labels and is not a fair comparison.

Why not just…?

Why not just use stronger augmentation or domain randomization? Augmentation also raises within-class variance on synthetic data, so it may produce exactly the effect we attribute to collapse control. This is the most serious threat. We treat augmentation strength as an axis of the measurement grid. If heavy augmentation alone reaches the same source $\mathrm{NC}_1$ and the same real accuracy as the entropy regularizer, the method claim fails. The diagnostic claim survives, since it would explain why augmentation works and give a label-free way to tune it. We will report that outcome plainly if it occurs.

Why not just run self-ensembling with a better confidence threshold? Self-ensembling's class-balance term assumes a roughly uniform target, which its authors acknowledge. Retuning the threshold does not change what the source backbone has discarded. If collapsed source features merge visually distinct real images of one class, no threshold recovers them. The comparison between self-ensembling on a standard source backbone and on our uncollapsed one, run on identical data, measures this directly.

Why not just normalize features to the unit sphere, or rescale them radially? Unit normalization is essentially free, and DSGK shows spherical geometry is competitive. A Geometry-Based View of Mahalanobis OOD Detection further shows that a partial rescaling $z/\|z\|^\beta$ with $\beta<1$ beats full normalization. Normalization changes feature scale but does not restore within-class spread lost during training. We include the sphere head and a radially rescaled Euclidean head in the geometry comparison. If they match our method at equal source $\mathrm{NC}_1$, the conclusion is that geometry choice is secondary to collapse control.

Proposed Experiments and Decisive Checks

The first decisive check is predictive. Across at least 60 checkpoints per benchmark, the source and target $\mathrm{NC}_1$ pair should rank real accuracy better than single-sided $\mathrm{NC}_1$, local intrinsic dimension, or target entropy. Its checkpoint-selection regret should fall below that of selecting on synthetic validation accuracy. If source $\mathrm{NC}_1$ shows no inverted-U or monotone relation with real accuracy, the core hypothesis is wrong.

The second check is the method on VisDA-C, measured against our own re-run of self-ensembling. We report per-class accuracy, because the claim predicts the largest gains on classes whose renderings are most homogeneous. We then repeat the comparison on GTA5→Cityscapes for dense prediction, reporting mIoU on rare classes such as rider and motorcycle.

The third check is the curvature test at matched collapse described above. All runs use three seeds and shared compute budgets. The result is informative whichever way it falls: either collapse explains a good part of the sim-to-real gap and of hyperbolic gains, or it does not and curvature carries independent value.