Small segmentation networks fail when a hospital changes scanners or colonoscopes. They latch onto the texture, noise and color signature of the training site, and their accuracy falls at a new site. On AMOS, MRI organ segmentation drops by more than 20% mean Dice on scanners not seen in training (AMOS). Large medical vision-language models, which learn by matching images to their clinical text, are a tempting source of knowledge for these small networks. But copying everything such a model computes also copies its sensitivity to scanner texture, and these models were trained to match whole images to whole reports, so their per-pixel features are poorly tied to specific words.
This note proposes a different filter for distillation. Distillation here means training a small "student" network to imitate a large "teacher." We would distill only the part of the teacher's per-pixel features that lies in the span of text embeddings for anatomy and findings, and discard everything else. The bet is that scanner artifacts have no words. Glare, reconstruction kernels and vendor noise are not described in radiology reports, so the language-spanned part of a feature should carry anatomy and leave out much of the acquisition signature. This filter needs no target-site data and no labels beyond the usual source masks. It can also be tested cheaply before any student is trained.
Core Claim
Let $f_t(p)\in\mathbb{R}^D$ be the teacher's unpooled feature at pixel $p$, read from the final spatial map as in DenseCLIP. Let $T\in\mathbb{R}^{D\times K}$ stack the text-encoder embeddings of a bank of $K$ prompts. The bank covers organ or lesion names, shapes and boundary descriptors, written as several paraphrases each. Let $P_T = T(T^\top T)^{-1}T^\top$ be the projection onto their span. A small learned map $W$ lifts student features $f_s(p)$ into the teacher's space. The student is trained on labeled source images with its normal segmentation loss plus
$$\mathcal{L}_{\text{TSD}} = \sum_{p} \big\| P_T\, W f_s(p) - P_T\, f_t(p) \big\|_2^2 ,$$
so whatever the teacher encodes outside the text span is never imitated.
The claim has two parts, and both can be falsified. The first part is diagnostic. Within the text span, the teacher's features predict acquisition site worse than a random subspace of the same dimension does, while still predicting anatomy. The second part concerns the student. Students trained with this loss generalize to unseen sites better than students trained with full-feature imitation, with existing selective-distillation masks, or with a random-subspace version of the same loss.
We also expect the benefit to depend on how locally aligned the teacher is. This is the link to the global-local alignment gap, the finding that image-text matching on pooled vectors leaves individual pixel features weakly tied to words. A teacher whose pixel features actually line up with words should help more than one that was only ever matched to whole reports.
Relation to Prior Work
Existing selective distillation chooses what to transfer by where the teacher is active or what the task loss needs. DCSF-KD weights channels by pooled teacher activations. MasKD learns attention masks over "pixels of interest" restricted to locations both networks can represent. The detection version of GKD weights features by task-loss gradients. All three select along spatial or channel axes based on importance. None selects feature directions based on whether they survive a change of scanner, and none uses language. These methods supply our strongest unimodal baselines.
Medical distillation for robustness motivates the proposal. SIKD trains a teacher on images flattened to per-class mean intensities, removing texture, and distills its features into a student by uniform MSE. Its M&Ms gain from 68.6% to 78.5% Dice is strong evidence that texture drives cross-scanner failure. Building those images, however, requires full masks, and the student imitates every teacher dimension. The medical GKD seeks domain invariance through autoencoder alignment and contrastive graphs that are costly at the global level. Our filter is cheap, requires no masks to define, and removes texture by projection rather than by rebuilding the input.
The closest work is MEDALIGN. It distills from a frozen UniMed-CLIP by matching patch-similarity matrices and attention maps. It is evaluated only on report generation and question answering, with no scanner shift, and it transfers the teacher's full patch geometry. We transfer only the text-spanned part, measure the effect on dense segmentation across sites, and use MEDALIGN-style similarity matching as a baseline.
Two papers constrain the design. The first is Anatomy-VLM. It shows that globally trained medical vision-language models confuse anatomical regions, and that region-level alignment fixes this. Yet its frozen encoder reaches only 0.153 Dice on SIIM-ACR, so dense text maps from such teachers are too weak to use as pseudo-labels. That is why we use the text span as a filter on features, with source masks still providing supervision, rather than as the segmentation target. The second is INPUTADAPTER, which found that successful adaptation can move target data further from the source in Wasserstein distance. That warns against judging robustness by distribution distance, so our diagnostic measures how well a probe can predict the site, not how close the distributions are.
Research Program
The first stage is a diagnostic that gates the rest. On multi-site data with site labels, namely PolypGen's six centers and the M&Ms vendors, we extract frozen teacher pixel features. We then fit linear probes for site and for anatomy class on three versions of those features: the text-projected component, the residual, and random subspaces of matched dimension $K$. We sweep $K$ by growing the prompt bank. Teachers are UniMed-CLIP, the frozen teacher used in MEDALIGN, together with any locally aligned medical vision-language model we can obtain. For chest data we use Anatomy-VLM features on CheXmask, which tests a teacher trained for region-level alignment. If the text-projected component is no worse than random at predicting site, the hypothesis fails, and we stop and report that negative result.
The second stage trains real-time students under single-source domain generalization, meaning they see one training site and no target images. The student is a ResNet-backed segmenter of roughly TransNetR size. Training uses the segmentation loss plus $\mathcal{L}_{\text{TSD}}$, and the prompt bank is fixed before any target data is examined.
The third stage tests the global-local question directly. We correlate each teacher's zero-shot pixel-text localization quality on source data with the cross-site gain its student receives.
Baselines
The naive reference is the same student trained only with Dice and cross-entropy on the source site, with standard augmentation. Every gain is measured against it.
Full-feature imitation (MSE on all teacher dimensions through the same $W$) tests whether selection matters at all. MEDALIGN-style matching of patch-similarity matrices from the same teacher represents the closest existing vision-language distillation recipe.
The strongest existing competitors depend on the dataset. On cardiac MRI it is SIKD, which reports Dice rising from 68.6% to 78.5% on M&Ms. On polyps, the reference is the cascaded DPN68x2 of Double Encoder-Decoder Networks, which reports 76.19% Dice on unseen ETIS-LaribPolypDB. On PolypGen, TransNetR reports only a relative 2.81% mIoU margin. For AMOS cross-scanner MRI, the file contains no absolute baseline number, only the reported drop of more than 20%. We also run MasKD and DCSF-KD as unimodal selective distillers using the same teacher.
Because published protocols differ, we re-run SIKD, MasKD and DCSF-KD ourselves rather than comparing against published numbers. Every method gets the same student, data split, epoch count and number of hyperparameter trials, and models are selected on source validation only. Beating a baseline means a higher mean Dice across held-out sites with non-overlapping seed intervals, at equal inference cost.
The ablation that isolates the key ingredient is a random-subspace version of $\mathcal{L}_{\text{TSD}}$ with matched $K$. A second is an "invariance-without-language" selector, which projects onto the teacher feature directions that change least under strong photometric augmentation. An adapter-tuned teacher in the style of side adapter networks (Xu et al.) serves as a non-deployable upper bound.
Why not just…?
Why not just fine-tune the vision-language model on the segmentation task? It may be accurate, but it is not deployable. Real-time polyp work targets above 30 frames per second (TransNetR). DenseCLIP also found that fine-tuning CLIP directly underperforms language-guided adaptation by 3.9 mIoU on ADE20K. We include the adapter-tuned teacher as an upper bound to measure how much of that ceiling a small student recovers.
Why not just use SIKD and strip texture from the input? SIKD is a real threat on cardiac MRI. It needs full masks to build its training images and imitates every dimension uniformly, but it may still win there. Running both methods, and their combination, on M&Ms settles the question. If SIKD alone matches us, our contribution narrows to settings where masks are partial or absent.
Why not just use heavy intensity and color augmentation? Augmentation covers the variations someone thought to simulate. Reconstruction-kernel and vendor effects are hard to script, and AMOS's more-than-20% MRI drop persists under standard pipelines. The augmentation-invariance selector tests the strongest form of this alternative. If it equals text-subspace distillation, language adds nothing, and we will say so.
Why not just select directions with low between-site variance? That works only if unlabeled multi-site data exists at training time, and single-source generalization rules it out. We still run it as an oracle on PolypGen to see how close language-based selection comes.
Decisive Checks
The experiments that decide the proposal are the following. The diagnostic must show that the text span predicts site worse than random subspaces of the same size. Students must be evaluated in two settings: Kvasir-SEG to PolypGen, ETIS-LaribPolypDB and CVC-ColonDB; and M&Ms leave-one-vendor-out together with AMOS MRI on unseen scanners, with per-organ results for pancreas, adrenal glands and duodenum, where shift does the most damage. In both settings, text-subspace distillation must beat both the random-subspace control and the augmentation-invariance control.
Finally, the student's gain should track the teacher's local alignment quality. If a globally trained teacher helps as much as a region-aligned one, the global-local framing is wrong even if the method works.
References
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting
- DCSF-KD: Dynamic Channel-wise Spatial Feature Knowledge Distillation for Object Detection
- Masked Distillation with Receptive Tokens
- Gradient-Guided Knowledge Distillation for Object Detectors
- Shape-intensity knowledge distillation for robust medical image segmentation
- Exploring Generalizable Distillation for Efficient Medical Image Segmentation
- Enhancing Medical Large Vision-Language Models via Alignment Distillation
- Anatomy-VLM: A Fine-grained Vision-Language Model for Medical Interpretation
- Adapt the Data, Not the Model: Input-Space Adaptation for Frozen Time-Series Predictors
- Double Encoder-Decoder Networks for Gastrointestinal Polyp Segmentation
- TransNetR: Transformer-based Residual Network for Polyp Segmentation with Multi-Center Out-of-Distribution Testing