A clinical model trained on a few hundred chest X-rays from a handful of hospitals often learns which hospital an image came from rather than what disease it shows. The site's scanner, its text markers and its patient mix happen to correlate with the label, and a classifier with enough capacity will use them. Foster, Kakade and Salakhutdinov argued that the effective sample size for generalizing to a new domain is the number of independent domains, not the number of images. They turned that argument into a simple rule: keep a feature only if its predictive effect is statistically consistent across training domains. Their rule worked on raw features of linear models and stopped there. This note proposes the deep-latent version. We take a frozen pre-trained encoder, align each training site's embeddings into a shared frame in closed form, and then fit the classifier only along latent directions whose class-separating effect is stable from site to site, as measured by a $t$-statistic computed across sites.
The opportunity matters because the regime it targets is common and poorly served. In this regime there are several sites, few labeled examples per site, no large pool of unlabeled target data, and a backbone that cannot be retrained. The strongest cross-domain few-shot methods either need unlabeled target images during training or modify pre-training. Fine-tuning larger models makes overfitting worse when data is scarce. The method we propose is a single closed-form computation on top of frozen features. It is cheap to test, and if it fails it fails clearly.
Core Claim
When labeled data come from $m$ sites with few examples each, a linear readout on frozen embeddings should be built from directions whose discriminative effect has a large mean relative to its between-site spread. Pooled discriminative strength is the wrong criterion. That criterion has a closed-form solution, but it only behaves well if the sites' latent spaces are first put into a common coordinate frame. The claim is that this combination generalizes to an unseen site better than pooled regularized logistic regression and better than per-site standardization alone. The advantage should be largest when a site-specific shortcut correlates with the label inside training sites. It should grow with $m$ and shrink to zero when $m$ is very small.
Method
The first stage is alignment. For each training site $j$, we center and scale its embeddings with site-specific statistics. This is the standardization step that Latent Space Translation via Semantic Alignment uses before stitching independently trained models; there it estimates angle-preserving maps from shared anchor points. We use class means as the anchors. For each site we solve an orthogonal Procrustes problem that rotates its class-mean matrix onto the pooled class-mean matrix. Orthogonal Procrustes is the best rotation between two point sets, computed by a single singular value decomposition. With few examples per class, a rotation is the most flexible map we trust. Least-squares or affine maps would overfit, and EASE shows that closed-form projections are both fast and adequate at this sample size. We L2-normalize all features, following the evidence from A Comparison of Machine Learning Methods for Cross-Domain Few-Shot Learning that L2 normalization strongly helps simple classifiers on frozen features. We also follow the norm-bias argument in Spherical Procrustes Alignment, which notes that small datasets push networks to inflate embedding length instead of improving direction.
The second stage is stability filtering. For a class contrast $c$, let $w_j \in \mathbb{R}^d$ be the shrunken within-site discriminant vector of site $j$ in the aligned frame: the class-mean difference multiplied by the inverse of the site's shrunken within-class covariance. Let $\mu$ be the average of the $w_j$ across sites and $S$ their across-site covariance. For a unit direction $v$, the across-site $t$-statistic of its effect is proportional to $v^\top \mu / \sqrt{v^\top S v / m}$. Maximizing it gives
$$v^\star \propto (S + \lambda I)^{-1}\mu,$$
where $\lambda > 0$ is a shrinkage constant. Shrinkage is required because $S$ has rank at most $m-1$. We set $\lambda$ analytically with Ledoit–Wolf shrinkage rather than by cross-validation, since validation on tiny sets is noisy enough to choose worse settings than defaults (A Comparison of Machine Learning Methods for Cross-Domain Few-Shot Learning). When $S \to 0$, meaning every site agrees, $v^\star$ reduces to the pooled discriminant direction. When a direction's effect flips sign between sites, it is suppressed even if its pooled effect is large.
For multi-class problems we take the top-$r$ generalized eigenvectors of the pair $(\sum_c \mu_c \mu_c^\top,\; S_{\text{pooled}} + \lambda I)$. This gives an $r$-dimensional subspace, and inside it we fit a cosine-similarity classifier. A new site is processed in one of two ways. In the label-free path, we standardize it with its own unlabeled statistics, using only the deployment batch. In the few-shot path, it also gets a Procrustes rotation from its support set.
The method plays three established ideas against each other. From Foster et al. it takes the domain-count statistic and extends it from raw features to rotated latent directions. A rotated basis is necessary because a $t$-statistic depends on the coordinate system, and unaligned latent coordinates mean different things at different sites. From Latent Space Translation it takes closed-form angle-preserving alignment. From the clinical failure documented in Unveiling COVID-19 from Chest X-ray with deep learning, where models clustered by hospital and reached sensitivity 0.54 and specificity 0.80 on the CORDA cohort, it takes the specific failure mode the method is meant to resist.
Research Program
The prior work falls into three groups, each playing a different role here.
The first group motivates the proposal. Domain Adaptation: Overfitting and Small Sample Statistics supplies the domain-count argument and the T-greedy method. Those authors leave the extension to deep latent representations explicitly open, and their method needs at least three domains; our method inherits that requirement. The COVID chest X-ray study and Towards AI-driven Next Generation Personalized Healthcare and Well-being, which describes private clinical cohorts of fewer than 100 samples, define the regime.
The second group constrains the design. Cross-Level Distillation and Feature Denoising assumes unlabeled access to 20% of the target dataset during training. Dynamic Distillation Network assumes an unlabeled target stream and reports only modest gains on ChestX. We assume neither. CPLSR improves BSCD-FSL by reserving latent space during pre-training, which rules it out for fixed foundation models; we never touch pre-training. Few-Shot Adversarial Domain Adaptation aligns domains by training an adversarial pair discriminator on 1 to 7 target samples, and it concedes that weight sharing breaks down under extreme shift. We replace the adversarial game with closed-form rotations on a frozen backbone. Partial disentanglement for domain adaptation separates invariant from site-specific latent subspaces using variational autoencoders and normalizing flows. That approach needs a hand-tuned style dimension and abundant data. Our filter separates the two kinds of subspace with a statistic instead of a generative model, and needs no subspace size beyond $r$.
The third group supplies methods we reuse: Procrustes alignment from Latent Space Translation and SPA, and normalization from the shallow-classifier study. GeRA shows that, with few pairs, a geometric regularizer stops alignment from destroying pre-trained structure. Our use of rotations only is a cruder version of the same caution.
The work has three parts. First, we build the estimator and a synthetic controlled-shortcut generator. Second, we analyze how the out-of-site risk of $v^\star$ scales with $m$. A direction's standard error falls like $1/\sqrt{m}$, so we expect the gap between stable and spurious directions to become detectable only once $m$ is around 4 to 5. Third, we test on real multi-domain and clinical data.
Baselines
The naive reference point is pooled training. We concatenate all sites' frozen, L2-normalized embeddings and fit an L2-regularized logistic regression with default regularization. A Comparison of Machine Learning Methods for Cross-Domain Few-Shot Learning found this setup, together with cosine classifiers, consistently beat more complex classifiers on frozen features in BSCD-FSL, so it is a strong baseline rather than a weak one. It uses exactly the same features and labels as our method.
The second baseline is per-site standardization followed by the same logistic regression. It isolates how much of any gain comes from moment alignment alone. This connects to the transductive batch-statistic updates that A Broader Study of Cross-Domain Few-Shot Learning found consistently helpful.
The third baseline is the strongest existing alignment method in this regime. We use few-shot adversarial adaptation in the style of FADA, with its domain-class pair discriminator trained on the frozen features and given the same labeled examples. We also run a T-greedy feature-selection baseline that applies Foster et al.'s statistic to unaligned raw embedding coordinates.
The key ablations remove one ingredient at a time. One removes the Procrustes rotation, which tests whether alignment is needed for the statistic to be meaningful. Another replaces $(S+\lambda I)^{-1}$ with the identity, which recovers the pooled discriminant and tests whether stability filtering matters. A third replaces analytic shrinkage with cross-validated $\lambda$.
For fairness, every method uses the same frozen backbone (a self-supervised DINO ViT, since CPLSR reports that frozen DINO reaches 52.73% average 1-shot and 64.67% average 5-shot on BSCD-FSL, close to the best adapted methods). Every method also gets the same per-site labeled budget, the same leave-one-site-out splits and a matched hyperparameter budget, with default settings when tuning is infeasible.
There are reference numbers in neighboring settings. iMSDA reports 89.22% on PACS with Sketch as the target and 76.19% average on Office-Home, but it uses full unlabeled target data, so it serves as an upper reference rather than a competitor. On MNIST, Foster et al. report that T-greedy keeps AUROC above 0.8 while standard greedy selection falls toward 0.5. For our exact setting, leave-one-domain-out evaluation with k labeled examples per class per site on frozen embeddings, no reported result was found on Lacuna. Beating the baselines means a higher mean worst-site and mean held-out-site accuracy than standardized logistic regression, with 95% confidence intervals over at least 50 resampled splits that do not overlap zero difference.
Why not just…?
Why not just pool everything and fit regularized logistic regression? This is the real threat. On clean benchmarks where sites differ only in style, we expect it to tie with our method, because there the stable direction and the pooled direction coincide. The controlled-shortcut experiment below is designed to separate them. In it, a feature correlates with the label within training sites but with a sign or strength that varies by site. Pooled training cannot detect that variation, because it only sees the averaged effect. If pooled training matches us even there, the idea has failed.
Why not just standardize each site and stop? Moment alignment removes site offsets and scale differences. It does not remove a direction whose class effect is site-dependent, because that information lives in conditional statistics, not marginal ones. The ablation that removes the stability filter but keeps standardization measures exactly this. If the gain from the filter is within noise across all experiments, we will report that alignment was the whole story.
Why not just fine-tune the backbone, or use a distillation method like CLDFD? Fine-tuning hits the capacity trap documented in the COVID chest X-ray study, where ResNet-50 overfit more than ResNet-18. Distillation methods require unlabeled target corpora that our regime lacks by construction. We will still include end-to-end fine-tuning at a matched labeled budget as a check.
Why not just apply T-greedy to raw embedding coordinates? Embedding coordinates are not aligned across sites, so a coordinate's effect can look unstable purely because of rotation between sites. The unaligned T-greedy baseline tests whether our alignment step is necessary.
Proposed Experiments
The first experiment is a controlled shortcut. We use PACS and Office-Home as multi-site testbeds, with $m$ ranging from 3 to 5 training domains and $k \in \{1,5,10\}$ labeled examples per class per domain. We test on the held-out domain. In a second condition, we overlay a synthetic marker, such as a corner text tag, whose correlation with the label is positive in some training domains and absent or reversed in others. This mimics the hospital-annotation artifacts in the COVID study. The decisive prediction is that pooled and standardized logistic regression lose accuracy under the shortcut while $v^\star$ does not. We will also probe whether $v^\star$ places near-zero weight on a linear probe trained to detect the marker.
The second experiment scales the number of domains. We replicate Foster et al.'s MNIST "2 versus even" to "odd" construction on deep embeddings, and we vary $m$ by splitting training data into artificial sites. The prediction is that our advantage appears at $m \ge 4$ and that, at $m=3$, analytic shrinkage collapses $v^\star$ toward the pooled discriminant rather than hurting performance.
The third experiment is clinical. We construct multi-site chest X-ray splits, with CORDA as one site and other public cohorts as further sites, holding out one site at a time. We report sensitivity and specificity against the CORDA reference of 0.54 and 0.80, together with expected calibration error, since SPA shows small medical datasets badly miscalibrate.
Finally, BSCD-FSL serves as a single-target sanity check where only one source domain exists. There the method reduces to the aligned pooled classifier, and it should neither help nor hurt relative to frozen DINO.
The proposal is falsified if, under the controlled shortcut, the stability filter adds no held-out-site accuracy over per-site standardization.
References
- Latent Space Translation via Semantic Alignment
- EASE: Unsupervised Discriminant Subspace Learning for Transductive Few-Shot Learning
- A Comparison of Machine Learning Methods for Cross-Domain Few-Shot Learning
- Spherical Procrustes Alignment for Reliable Medical Audio Diagnosis
- Unveiling COVID-19 from CHEST X-Ray with Deep Learning: A Hurdles Race with Small Data
- Domain Adaptation: Overfitting and Small Sample Statistics
- Towards AI-driven Next Generation Personalized Healthcare and Well-being
- Cross-Level Distillation and Feature Denoising for Cross-Domain Few-Shot Classification
- Dynamic Distillation Network for Cross-Domain Few-Shot Recognition with Unlabeled Data
- Cross-Domain Few-Shot Learning with Coalescent Projections and Latent Space Reservation
- Few-Shot Adversarial Domain Adaptation
- Partial disentanglement for domain adaptation
- GeRA: Label-Efficient Geometrically Regularized Alignment
- A Broader Study of Cross-Domain Few-Shot Learning