When a model is trained on data pooled from several hospitals, scanners, or photo styles, part of its internal representation ends up recording where an example came from rather than what it shows. When the model meets a new site, those "site directions" drift, and accuracy falls. Current fixes at deployment time act on everything at once. They either re-estimate all normalization statistics, or they decorrelate or whiten the whole feature space. On real, natural shifts these blanket corrections often hurt, because they also move the directions that carry the label. We propose a narrower fix. Use the diversity of the training sources to find the few feature directions that vary across sources but not across classes. Then, at test time, re-standardize the incoming stream only inside that subspace, in closed form and without backpropagation, and leave every other direction untouched.
The opportunity comes from two findings that pull in opposite directions. Medical work shows that forcing full domain invariance throws away predictive signal (Embracing the Disharmony in Heterogeneous Medical Data). Test-time adaptation (adjusting a trained model on unlabeled deployment data as it arrives) does well on synthetic corruptions but often fails on natural shifts (Benchmarking Test-Time Adaptation against Distribution Shifts in Image Classification). Our bet is that both failures have one cause: the correction is not targeted. Having many labeled training sources is exactly what lets us target it.
Core Claim
Multi-source training data contains a supervision signal that existing adaptation methods ignore. Take two examples of the same class from different sources. The difference between their features is evidence of a site direction, a direction that is redundant with respect to the label. We estimate these directions from labeled multi-source data before deployment. We claim that re-standardizing the target stream only within this "site subspace" will beat full-space correlation alignment and full batch-norm re-estimation on natural multi-site shifts. We further claim that the gain will grow with the number and diversity of training sources. This links the diversity direction, which supplies the signal, to harmonization, which is the operation, to redundancy, which is the object being removed, and to domain shift, which is the regime.
Concretely, let $z \in \mathbb{R}^d$ be a penultimate-layer feature. Let $\mu_{s,c}$ be the mean feature of class $c$ in source $s$, and $\mu_c$ the mean of class $c$ pooled over sources. We form two matrices. The first is the within-class, between-source scatter $S_{\text{site}} = \sum_{s,c} p_{s,c}(\mu_{s,c}-\mu_c)(\mu_{s,c}-\mu_c)^\top$, where $p_{s,c}$ is the fraction of training data in that cell. The second is the between-class scatter $S_{\text{label}}$ of the $\mu_c$. The site subspace $U \in \mathbb{R}^{d\times k}$ is spanned by the top $k$ solutions of
$$S_{\text{site}}\, u = \lambda\,(S_{\text{label}} + \epsilon I)\, u,$$
which gives directions where source variation is large relative to class variation ($\epsilon$ is a small ridge term). We also extend $S_{\text{site}}$ with within-class between-source covariance differences, so that site effects on spread are captured as well as effects on means.
At test time we maintain a running estimate of the mean $m_t$ and covariance $C_t$ of the target stream projected onto $U$, stabilized with a statistics memory in the spirit of MemBN. We map those statistics onto the pooled-source statistics $(m_0, C_0)$ in $U$:
$$z' = z + U\left(C_0^{1/2} C_t^{-1/2}\,(U^\top z - m_t) + m_0 - U^\top z\right).$$
The orthogonal complement of $U$, where the label lives, is never touched. Every quantity is computed from features, so the method applies equally to classification and to regression.
Relation to Prior Work
Several papers motivate the design. FRET shows that off-diagonal feature covariance grows with shift severity and removes it at test time. It decorrelates the whole space from a single source model, and its simple variant S-FRET breaks under target label shift. We instead decorrelate only along directions certified as label-irrelevant by multi-source labels. Gaussian Image Anomaly Detection with Greedy Eigencomponent Selection finds that high-variance principal components often encode background and domain nuisance rather than signal. That supports our premise, but it selects directions by search with no notion of source identity. Embracing the Disharmony argues against full invariance and bridges sites with auxiliary demographic tasks. It needs demographic metadata and offline fine-tuning; we need only source identity and work online.
Other papers supply the method. Test-time Correlation Alignment aligns target covariance to a pseudo-source in closed form, at a tiny fraction of usual compute. We reuse its closed-form machinery but restrict it to a learned subspace. That restriction is the whole hypothesis. MemBN supplies the memory queue for stable small-batch statistics.
Other work constrains the design. The PredBN paradox in the TTA benchmark rules out blanket statistic re-estimation on natural shifts. WILDS 2.0 finds that training-time invariance methods (DANN, CORAL) rarely beat plain training on real shifts, which argues against folding the fix into training. MeTA also uses multiple sources at test time, but as separate black-box models blended by output entropy. It never inspects internal features and requires one model per source. We use one pooled model and use the sources only to define geometry. The bound in Analysis of Representations for Domain Adaptation frames the trade-off we exploit. Reducing source–target divergence helps only if the joint-error term does not grow, and confining the change to label-irrelevant directions is a way to protect that term.
Research Program
The work has three parts.
Measurement. On pooled multi-source training, we measure how much target-minus-source feature drift falls inside $U$ versus its complement. We do this across natural shifts and synthetic corruptions. This tests the premise directly, before any accuracy claim.
Method. We build the closed-form site-subspace recalibration above. It runs on the penultimate layer. A second variant applies the same projection to the per-channel statistics of every normalization layer, which turns it into a subspace-restricted version of batch-norm re-estimation. The rank $k$ is chosen by leave-one-source-out validation, never on target data.
Diversity scaling. We vary the number of training sources and their stylistic spread. If source diversity is what identifies site directions, the estimate of $U$, and with it the gain, should improve as sources are added. This turns the "diversity over volume" claim from the data-centric survey into a measurable quantity for one specific mechanism.
Baselines
The naive reference point is the unadapted model trained by plain pooled empirical risk minimization (ERM) on all sources. On Camelyon17-WILDS, WILDS reports roughly 93% in-distribution and about 70% out-of-distribution accuracy for ERM, with 70.8% in WILDS 2.0. On PovertyMap the out-of-distribution Pearson correlation is 0.45. On Office-Home, the TTA benchmark reports a source-model accuracy of 59.8%.
The strongest direct competitor is LinearTCA/LinearTCA+ from Test-time Correlation Alignment. It is also closed-form and backprop-free, and it differs from our method only in aligning the full feature space against a pseudo-source. It reports a 4.95% accuracy improvement over the previous best method on CIFAR-10-C with a ViT backbone, and a 1.86% average gain for zero-shot CLIP across datasets including VLCS. No reported result was found on Lacuna for LinearTCA's absolute accuracy on Office-Home, PACS, DomainNet, or Camelyon17, so we rerun it.
The strongest reported online result on Office-Home is NRC at 69.6% (benchmark). NRC is a test-time domain adaptation method that refines predictions using each target sample's nearest neighbors. On CIFAR-100-C, MeTA reports 34.2% average error with multiple source models (MeTA). On Camelyon17, the offline Noisy Student result of 86.7% from WILDS 2.0 serves as an upper reference: it uses far more target data and retraining than our setting allows. We also run PredBN, Tent (Tent), S-FRET (FRET), and MemBN.
Fairness is enforced in four ways. All methods share the same pooled-ERM backbones, ResNet-50 and ViT-B/16, except MeTA, which receives per-source models trained with the same total compute. All methods see the same target streams, in the same order and at the same batch sizes. All hyperparameters are tuned under one leave-one-source-out budget with no target labels. Wall-clock time and memory are reported.
Three ablations isolate the key ingredient, which is the class-conditioned between-source subspace. The first replaces $U$ with the top-$k$ principal components of pooled features. The second replaces $U$ with between-source directions computed without class conditioning. The third uses a random $k$-dimensional subspace. Beating LinearTCA means a consistent gain in average target accuracy across held-out domains on Office-Home, PACS, DomainNet-126, and Camelyon17, or a gain in correlation on PovertyMap. The gain must exceed seed variance, and the method must not lose to the source model on any held-out domain where LinearTCA does lose.
Why not just…?
Why not just align the full covariance, as LinearTCA does? This is the real threat. If target drift on natural shifts is spread across the whole space, full alignment will match or beat us. The measurement study decides this. If the fraction of drift that falls inside $U$ is not substantially higher than its share of dimensions, the premise fails and we would report that. The case for restriction is indirect but specific. Blanket statistic correction degrades natural-shift accuracy (the PredBN paradox), and full decorrelation breaks under label shift (S-FRET). A label-shifted Office-Home stream, made by subsampling classes in the target, tests whether full alignment pays a price we avoid.
Why not just project out the top principal components? The eigencomponent-selection finding (Gaussian anomaly detection) suggests that high-variance directions are often nuisance, so this simpler heuristic could plausibly work. It is therefore an ablation, not a strawman. Our argument is that the largest-variance directions in a pooled model also include large between-class directions, since class structure is usually the dominant variance in a supervised embedding. Removing them should hurt. If top-PC removal matches our method, the multi-source labels are not earning their keep, and we would say so.
Why not just train with an invariance loss, or harmonize offline with ComBat-style correction? WILDS 2.0 reports that CORAL and DANN rarely beat ERM on real shifts. Embracing the Disharmony shows that preserving site variation lifted ADNI-to-PENN transfer from 78.84% to 92.14%. Offline invariance also cannot react to a site never seen in training. We include a CORAL-trained backbone as a baseline to confirm this on our splits rather than assume it.
Why not just keep one model per source and blend them, as MeTA does? MeTA is strong on corruption benchmarks (34.2% error on CIFAR-100-C), but it multiplies model storage by the number of sources and gives no handle on internal redundancy. We compare at matched total compute on CIFAR-100-C and on PACS with sources as domains.
Proposed Experiments
The primary benchmarks are natural multi-source shifts with known source identity. These are Office-Home and PACS (leave one domain out, the rest as sources), DomainNet-126, Camelyon17-WILDS (training hospitals as sources), and PovertyMap-WILDS (countries as sources; a regression task, where pseudo-labeling methods are known to struggle). iSTAGING MRI is a stretch target if access permits. CIFAR-10-C and CIFAR-100-C serve as a control where shift is synthetic.
There are four decisive checks.
- Premise check. The fraction of target drift lying in $U$ should predict, across held-out domains, how much our method gains over the source model.
- Diversity scaling. On DomainNet and PACS, gain should rise with the number of training sources, holding total training images fixed. This separates diversity from volume.
- Label shift. On class-imbalanced target streams we should degrade less than S-FRET and LinearTCA.
- Corruption control. On CIFAR-C with clean sources, the site subspace does not span the corruption directions, so we expect small gains, below LinearTCA. Seeing that confirms the method does what it claims rather than acting as a generic regularizer. Large gains there would instead suggest an unexplained effect we would need to investigate.
References
- Embracing the Disharmony in Heterogeneous Medical Data
- Benchmarking Test-Time Adaptation against Distribution Shifts in Image Classification
- MemBN: Robust Test-Time Adaptation via Batch Norm with Statistics Memory
- FRET: Feature Redundancy Elimination for Test Time Adaptation
- Gaussian Image Anomaly Detection with Greedy Eigencomponent Selection
- Test-time Correlation Alignment
- Extending the WILDS Benchmark for Unsupervised Adaptation
- CONTRAST: Continual Multi-source Adaptation to Dynamic Distributions
- Analysis of Representations for Domain Adaptation
- A Survey on Efficient Large Language Model Training: From Data-centric Perspectives
- WILDS: A Benchmark of in-the-Wild Distribution Shifts
- Tent: Fully Test-Time Adaptation by Entropy Minimization