When a segmentation model trained on labeled CT scans has to outline the liver in an MRI scan, and only one annotated MRI example is available, most current methods compress that example into a single average feature vector, a "prototype," and label each query pixel by its similarity to it. The average captures where the organ sits in feature space. It says nothing about how the organ's features spread: which directions vary a lot across patients, slices, and scanners, and which barely move. This proposal takes that spread, the covariance, from the data-rich source modality and moves it into the data-starved target modality. The one MRI example supplies the center of the organ's feature distribution. Many labeled CT patients supply its shape.
The idea rests on a finding from natural-image classification. Calibrating Biased Distribution in VFM-derived Latent Space via Cross-Domain Geometric Consistency shows that in large pretrained vision models (vision foundation models such as CLIP and DINOv2), a semantic class forms an ellipsoid of the same shape, with the same covariance eigenvectors and eigenvalues, across very different datasets. Transferring that shape to data-poor classes gave +17.1% on CIFAR-10-LT and +17.01% over FedAvg on CIFAR-100 under label skew. No one has asked whether the same holds for dense patch features of anatomy across CT, MRI, and X-ray, or whether it helps few-shot segmentation. That question is cheap to test, can fail in an informative way, and would give a simple method if it holds.
Core Claim
Our hypothesis is that, in the patch features of a frozen foundation model, an organ's feature covariance is largely preserved across imaging modalities even though its mean shifts. If so, a one-shot target-domain segmenter that models each class as a Gaussian, with its mean taken from the support image and its covariance shrunk toward the source-domain covariance of the same organ, should beat both mean-only prototypes and covariance estimated from the support image alone. It should also be competitive with the heavier structural matchers now leading cross-domain few-shot medical image segmentation.
The claim differs from its nearest neighbors in three ways. The GGEUR work transfers global class covariance for image-level classification on natural images, and it moves shape from head classes to tail classes, not from one modality to another. We apply it to dense per-patch features, and our transfer runs across modalities. The Riemannian prostate models, Geometric multimodal learning to support prostate cancer diagnosis on limited and multicentric bi-parametric MRI data and MFM-Geom, compute covariance matrices of features as a pooled classification descriptor. Their within-study covariance is the signal. For us, covariance is a transferable prior that corrects a biased one-shot estimate. Those papers do motivate the proposal: second-order statistics degraded far less than average pooling with 10% of the data (about 10% AUC loss against over 30%) and across unseen centers (5.4% against 10.4%). Finally, C-Graph, FAMNet and RobustEMD build invariance into how support and query features are matched: spatial graphs, mid-frequency Fourier bands, and texture-weighted optimal transport respectively. We keep matching deliberately simple and put the invariance into the class model.
Why the support image alone is not enough
A skeptic's first objection is that in dense segmentation even a single support slice contains hundreds of foreground patch tokens, so its covariance can be estimated directly. That is true, and it is why the proposal is framed as shrinkage rather than replacement. The tokens from one slice of one patient are strongly correlated. They sample variation within that slice, such as position within the organ and local texture. They do not sample the variation that actually separates support from query: a different patient, a different slice level, a different coil. The source domain holds many labeled patients and does sample that between-patient variation. If the between-patient part of the covariance transfers across modalities, then mixing it in should help in exactly the directions the support slice cannot see.
Method
Each image passes through a frozen foundation-model encoder to give patch features $f \in \mathbb{R}^d$. Features are reduced to $d \approx 64$–$128$ dimensions by a PCA fitted on source data, so the covariances stay well conditioned. On the source modality we compute, for each organ class $c$ and for background, a covariance $\Sigma^{src}_c$ pooled over all labeled patients. On a target query, each class gets a mean $\mu_c$ from the masked support tokens and a covariance
$$\hat\Sigma_c = (1-\alpha)\,\Sigma^{sup}_c + \alpha\, s_c\,\Sigma^{src}_c + \epsilon I,$$
where $\Sigma^{sup}_c$ is the support-token covariance and $s_c$ is a scalar that rescales the source covariance to the support's total variance. The scalar allows for a global contrast change while keeping the eigenvector directions and relative eigenvalues. The weight $\alpha$ is chosen on source-domain validation episodes only, and $\epsilon I$ is a small ridge term. Each query token is labeled by the Gaussian log-likelihood ratio between the class and background models. Coarse patch-level scores are upsampled with a fixed bilinear rule.
In a second variant, a small decoder is trained on source episodes only and receives these likelihood maps alongside the features. This isolates how much the covariance model contributes beyond boundary refinement.
When the target class has no counterpart in the source, as in the abdominal-to-chest-X-ray lung setting, we borrow the covariance of the source class whose mean is nearest to the support mean after centering each domain. The resulting error is reported separately, because this is where the assumption is weakest.
Research Program
The work has three stages. The first two stages, not the segmentation scores, determine whether the idea stands.
The first stage is a measurement study that needs no segmentation model. For each organ in CHAOS, which contains CT and MRI of the same organs, we compute class covariances in CT and in MRI features. We compare the leading eigenvectors by principal angles, and we compare the full covariances by the affine-invariant Riemannian distance to the covariance in the other modality. Both are set against two reference points: a different organ in the same modality, and a random rotation. We repeat this across encoders (DINOv2, CLIP, BiomedCLIP, and an ImageNet ResNet-50 as used by prior matchers) and across layers. The pre-registered outcome is that same-organ, cross-modality covariances must be measurably closer than different-organ, same-modality covariances. If they are not, the core hypothesis fails for anatomy, and we report that as the result.
The second stage splits the covariance into two parts. The within-slice part is estimated from single slices. The between-patient part is the covariance of per-patient means. We test which part transfers across modalities, which tells us whether the shrinkage story above is the right mechanism.
The third stage is the segmentation benchmark described below.
Baselines
The naive reference point is mean-only prototype matching on the same frozen encoder: cosine similarity to the masked-average support feature, which is the PANet-style design that C-Graph and FAMNet describe as collapsing under domain shift. This baseline asks whether the added covariance does anything at all.
The key ablation removes the transfer by setting $\alpha = 0$, using support-only covariance with the same ridge term. This is the real threat to the idea and isolates its ingredient. We also run a structure-free control that replaces $\Sigma^{src}_c$ with a covariance pooled over all classes. This tests whether any well-estimated covariance helps, or only the organ-specific one.
The strongest existing method is C-Graph. The file reports a mean Dice of 72.83% for abdominal CT to MRI, 73.10% on source CT, and 78.38% for lungs on chest X-ray from an abdominal-trained model (C-Graph). FAMNet is the strongest reported method for cardiac LGE to bSSFP at 71.58% Dice, against 61.44% for the next method, and reaches 65.79% on abdominal CT to MRI (FAMNet). RobustEMD reports gains of 3.66% and 9.64% over prior methods with a ResNet-50 on CHAOS MRI to BTCV CT, but the file gives no absolute Dice, so we will rerun it rather than quote it.
For fairness, we run C-Graph and FAMNet with their published protocol and backbone. We also run them with the same frozen foundation encoder as ours, so a win cannot come from features alone. All methods use identical source-training data, the same support-selection seeds (at least five draws of the support slice per fold), and equal hyperparameter search budgets on source-only validation. We report mean and spread across support draws.
Beating the strongest baseline means that the training-free Gaussian variant matches C-Graph within its support-draw variance, or that the decoder variant exceeds it on at least two of the three cross-modality or cross-sequence settings, without losing source-domain Dice. Independently of C-Graph, the method must beat the $\alpha = 0$ ablation by a margin larger than the support-draw spread. Otherwise the transfer claim is not supported.
Why not just…?
Why not just estimate the covariance from the support image's own pixels? This is the most serious alternative, and it may partly win. Hundreds of foreground tokens are enough to fit a shrunk covariance. Our answer is empirical. The $\alpha = 0$ ablation runs this exact alternative, and the second stage of the program tests whether between-patient covariance, which a single support slice cannot contain, is the part that transfers. If the chosen $\alpha$ comes out near zero, or the ablation matches the full method, we conclude that transfer adds nothing and say so.
Why not just augment and normalize intensities so a standard network generalizes? The evidence in the file is discouraging. Histogram equalization did not stop shortcut learning across chest X-ray cohorts (Unveiling COVID-19 from Chest X-ray with deep learning), and the CHAOS challenge saw Dice fall from about 0.97 to 0.88 across modalities with conventional pipelines (CHAOS Challenge). Randomized-convolution appearance scrambling does help registration (MAD), and C-Graph argues that appearance filtering costs source accuracy. We will add random-convolution augmentation during source training to the prototype baseline, to measure whether augmentation alone closes the gap.
Why not just fine-tune the encoder on the support example, for instance with a low-rank adapter? Low-rank adaptation needed 20 target volumes to reach 45.64% Dice on a hard coronary target (NA-UNETR), and one slice gives little to fit. Still, test-time adaptation is a fair competitor. We will run adapter fine-tuning on the support slice, with self-supervised flip consistency, under equal wall-clock time, and report where it overfits.
Why not just use a heavier structural matcher like C-Graph with a foundation-model backbone? That may well be stronger in absolute Dice, which is why we run it on the same encoder. The two ideas are compatible, since the transferred Gaussian can serve as C-Graph's node likelihood. We will report that combination if the standalone method is competitive, and treat covariance transfer as an ingredient rather than a replacement.
Proposed Experiments
Experiments follow the standard cross-domain few-shot protocol used by FAMNet and C-Graph. The settings are abdominal CT to MRI on CHAOS and BTCV (liver, kidneys, spleen) in both directions, cardiac LGE to bSSFP (left ventricular blood pool, myocardium, right ventricle), cross-institution prostate from SmartTarget to TCIA Prostate3T (bladder, central gland, rectum), and abdominal to chest X-ray lungs as a stress test for the case with no matching source class. Metrics are Dice and 95th-percentile Hausdorff distance, with 1-shot as the main setting and 5-shot as a secondary one.
The first decisive check is the stage-one covariance-similarity test, run before any segmentation. The second is the $\alpha = 0$ ablation. The third is a dose-response curve: as the number of source patients used to estimate $\Sigma^{src}_c$ grows from 1 to all, target Dice should rise and then plateau. A flat curve would mean the benefit comes from regularization, not from transferred geometry. The fourth is a per-organ correlation. Across organs and encoders, the stage-one covariance distance should predict the Dice gain over the prototype baseline. That correlation is the most direct evidence that geometric consistency, and not something incidental, drives the result.
References
- Calibrating Biased Distribution in VFM-derived Latent Space via Cross-Domain Geometric Consistency
- Geometric multimodal learning to support prostate cancer diagnosis on limited and multicentric bi-parametric MRI data
- A Geometric Multimodal Foundation Model Integrating Bp-MRI and Clinical Reports in Prostate Cancer Classification
- Contrastive Graph Modeling for Cross-Domain Few-Shot Medical Image Segmentation
- FAMNet: Frequency-aware Matching Network for Cross-domain Few-shot Medical Image Segmentation
- RobustEMD: Domain Robust Matching for Cross-domain Few-shot Medical Image Segmentation
- Unveiling COVID-19 from CHEST X-Ray with Deep Learning: A Hurdles Race with Small Data
- CHAOS Challenge - combined (CT-MR) healthy abdominal organ segmentation
- MAD: Modality Agnostic Distance Measure for Image Registration
- A Neighborhood Attention Transformer Network for Enhanced 3D Segmentation of the Left Anterior Descending Artery