Stopping Sim-to-Real Prototype Adaptation When the Class Geometry Starts to Collapse

A common way to transfer a classifier from synthetic images to real ones uses no real labels. The model labels the real images itself, using the nearest class "prototype" (the average feature vector of a class). It then recomputes the prototypes from those guessed labels and repeats. This loop is fragile. Accuracy on real data typically rises and then falls as the model confidently absorbs its own mistakes. With no labeled real data, nobody knows when to stop. Our proposal is that the moment to stop can be read directly off the geometry of the target prototypes, specifically off the point where they begin to collapse. When wrong guesses accumulate, classes get swallowed by dominant neighbors and the feature space flattens. A label-free measure of how many directions and distinct prototypes the real-image features still occupy should therefore peak near the accuracy peak.

The opportunity is practical. Synthetic-to-real benchmarks show how large the gap is. VisDA: The Visual Domain Adaptation Challenge reports a source-only model dropping from 99% on synthetic renders to 28% on real COCO images. Prototype adaptation such as Transferrable Prototypical Networks for Unsupervised Domain Adaptation (TPN) recovers much of that gap, but it runs for a fixed number of iterations chosen with hindsight. In deployment there is no hindsight. A stopping rule derived from collapse geometry would turn a tuned recipe into one that can be run unattended. The same signal could also choose the strength of anti-collapse regularizers, which have their own label-free tuning problem.

Core Claim

We claim that, in pseudo-label prototype adaptation across a large synthetic-to-real gap, the late decline in target accuracy is mainly a geometric collapse event. Classes merge, and target features lose effective dimensions. A smoothed collapse score computed only from unlabeled target features should therefore have its first maximum within a few percent of the iteration where true target accuracy peaks.

Concretely, at iteration $t$ let $P_t$ be the $K \times d$ matrix of centered target prototypes, where $K$ is the number of classes and $d$ the feature dimension. Each prototype is computed from pseudo-labels. Let $\sigma_i$ be the singular values of $P_t$, with $p_i = \sigma_i / \sum_j \sigma_j$. The prototype effective rank is

$$ R_t = \exp\Big(-\sum_i p_i \log p_i\Big), $$

which equals $K-1$ when the prototypes are spread evenly and falls toward 1 as classes merge. We pair it with the same quantity computed on the full target feature covariance. That second term catches the flattening of the whole feature space diagnosed by Tackling Dimensional Collapse toward Comprehensive Universal Domain Adaptation. The stopping rule is to halt at the first local maximum of a moving average of the summed log effective ranks.

The novelty is not detecting collapse, which singular-value spectra already do, nor stopping without validation data, which Early Stopping Against Label Noise Without Validation Data already does. The novelty is the claim, and its test, that in this regime the two are the same event, so collapse measurements double as a label-free stopping and model-selection criterion.

The claim is falsifiable. It fails if wrong pseudo-labels mostly shuffle members between clusters that keep their shape. In that case $R_t$ would stay flat while accuracy falls.

How This Relates to Prior Work

Three papers motivate the claim.

Early Stopping Against Label Noise Without Validation Data (Label Wave) shows a three-phase training trajectory under label noise and stops at the first minimum of epoch-to-epoch prediction changes. It is evaluated only on closed-set label noise and needs a visible "confusion" phase. Wrong pseudo-labels are structured, self-reinforcing noise, so that phase may be muted.

Layerwise Early Stopping for Test Time Adaptation (GALA) shows that indiscriminate test-time updates degrade accuracy. It chooses which layer to update using gradient alignment, and reports up to 15% gains on continual adaptation. It never looks at feature geometry.

Neural Collapse in Test-Time Adaptation links accuracy loss under shift to features losing alignment with their class weights. It notes that computing class prototypes online is hard and offers no stopping rule.

The untrained-prior literature supplies the analogy and a design principle. Methods such as JPEG Information Regularized Deep Image Prior for Denoising stop by monitoring a quantity that grows monotonically with memorized noise, in that case compressed file size. That method lifts PSNR from 21.74 dB unstopped to 27.34 dB on CBSD68. We look for the analogous quantity for pseudo-label noise in feature space.

Two papers constrain the design.

On Partial Prototype Collapse in the DINO Family of Self-Supervised Methods shows that prototypes can merge geometrically even when sample assignments look uniform. So counting occupied clusters is not enough, and we measure singular values of the prototypes themselves. The same paper also reports that forcing prototypes apart can hurt transfer to distant domains. That is a warning that a repulsion regularizer is not automatically safe under a synthetic-to-real shift.

ProtoTTA: Prototype-Guided Test-Time Adaptation minimizes the entropy of prototype similarities and reports stable long adaptation. That is a potential counterexample to the claim, and we address it directly by testing its restricted-update regime below.

TPN supplies the adaptation loop we instrument. VICReg and the uniformity loss of the dimensional-collapse paper supply the competing "prevent rather than detect" strategy.

Research Program

The first stage is diagnostic. We reimplement TPN-style pseudo-label prototype adaptation on a ResNet backbone and run it well past its usual budget. At every checkpoint we log five things: true target accuracy (for analysis only), per-class recall, $R_t$, the effective rank of the target feature covariance, the minimum pairwise prototype distance relative to within-class spread, and Label Wave's prediction-change count on target pseudo-labels.

The key analysis is whether accuracy decline coincides with class absorption. We expect to see one or two classes' recall going to zero while a neighbor's recall grows, with $R_t$ dropping at the same time. If it does not coincide, we have a clean negative result about what pseudo-label noise does to geometry.

The second stage turns the measurement into rules. One rule is a stopping rule. The other is a model-selection rule: among runs with different regularizer weights or learning rates, pick the run and checkpoint with the highest collapse score.

The third stage asks whether anti-collapse regularization makes stopping unnecessary. We add a target-side uniformity loss and, separately, KoLeo-style prototype repulsion, then check whether the accuracy curve still peaks.

Baselines

The naive reference is source-only training, which VisDA reports at 28% for AlexNet. We will rerun source-only on our own backbone rather than quote that figure. Next to it sits fixed-iteration TPN, run with the authors' schedule. Transferrable Prototypical Networks for Unsupervised Domain Adaptation reports 80.4% single-model accuracy on VisDA 2017. This is the strongest prototype-based competitor in our sources and the number our reproduction must approach before any comparison counts. The VisDA challenge winner's 92.8% relied on heavier pseudo-labeling pipelines. We treat it as context, not as a like-for-like baseline.

The decisive competitors are other label-free stopping rules applied to the identical training run, so every rule picks a checkpoint from the same trajectory. These are:

  • Label Wave's prediction-change minimum.
  • Lowest source validation loss, the rule practitioners actually use.
  • Lowest mean target prediction entropy.

The upper reference is an oracle that picks the best checkpoint using real target labels. "Beating" the field means two things. Our rule's chosen checkpoint must be closer to the oracle (lower regret, measured in accuracy points) than every other label-free rule, averaged over at least five seeds. It must also improve on fixed-iteration TPN when the training budget is deliberately overextended.

Two ablations isolate the ingredients. One uses only the full-feature effective rank, which is the dimensional-collapse diagnostic alone. The other uses only prototype effective rank. Their comparison shows whether prototype-level geometry adds information. All methods share data, backbone, augmentation, and compute. Any tunable smoothing window is fixed on Digits and transferred unchanged to VisDA.

Why not just…?

Why not just add a uniformity or VICReg-style loss and train to a fixed budget? This is the most serious threat. Tackling Dimensional Collapse toward Comprehensive Universal Domain Adaptation reports H-score rising from 48.1% to 79.8% on extreme Office-31 with exactly such a loss. If regularized adaptation shows no accuracy peak on VisDA, stopping is moot. We would report that outcome as the main finding rather than argue around it. Two reasons make us expect stopping to remain useful. First, that paper's loss weight is tuned by hand, which is the same hidden validation problem in another form. Second, the KoLeo-proto paper reports degradation from repulsion on distant domains. Stage three settles it by comparing regularized-fixed-budget, regularized-plus-stopping, and the collapse score used to select the regularizer weight.

Why not just use Label Wave on target pseudo-labels? It is cheap and already validated, landing within 0.2 to 0.9 points of the best accuracy under label noise. However, it was designed for random label flips that cause visible prediction churn. Self-reinforcing pseudo-labels may change slowly and smoothly, giving no clear minimum. Running both rules on identical trajectories answers this directly. If Label Wave matches our regret, the collapse score is unnecessary and we will say so.

Why not just stop on source validation loss? VisDA's own numbers show that synthetic accuracy is near-saturated while real accuracy sits far below. Source loss is therefore likely to keep improving long after target accuracy turns. This is included as a baseline precisely because it is the default.

Why not just restrict updates to normalization layers, as ProtoTTA does? If restricted capacity removes the peak, as the subspace-restriction results in untrained priors suggest, detection is unneeded. We will run TPN with normalization-only updates. If the curve plateaus without decline at accuracy comparable to full adaptation, that regime does not need our rule, and we will report the accuracy cost of the restriction.

Proposed Experiments

The primary benchmark is VisDA 2017 classification: 12 classes, about 152k synthetic CAD renders adapted to about 55k real COCO crops. The Digits transfers MNIST→USPS and USPS→MNIST, where TPN reports 92.1% and 94.1%, serve as a fast setting for fixing the smoothing window.

As a stress test of generality we will apply per-class prototypes to SynLiDAR→SemanticKITTI segmentation. There, Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation reports 15.2% mIoU source-only and 22.8% with adversarial masking, against a 55.5% fully supervised oracle. Long-tailed classes there should make class absorption especially visible.

The metrics are:

  • Target accuracy or mIoU at the selected checkpoint.
  • Regret relative to the oracle checkpoint.
  • Iteration distance between the signal's peak and the accuracy peak.
  • Across seeds and regularizer settings, Spearman correlation between collapse score and target accuracy.

The decisive check is the coincidence test from stage one. If, on VisDA, the drops in $R_t$ do not precede or match the accuracy decline in a majority of seeds, the core claim is wrong for synthetic-to-real pseudo-labeling. In that case, the per-class recall logs will show which other failure mechanism dominates.