A diagnostic model trained at one hospital usually gets worse at the next. The new site has different scanners, protocols and patients, but it does have plenty of unlabeled images. A standard way to use those images is label propagation: connect each image to its most similar neighbors, then let labels flow from labeled source images to unlabeled target ones. The difficulty is deciding what counts as "similar." In the raw feature space of an image network, scans from the same machine tend to sit near one another whatever the disease, so propagation can carry scanner identity instead of diagnosis. We propose building the neighbor graph in a different space: the scores of named clinical concepts such as "consolidation," "pleural effusion" or "cardiomegaly," produced by a medical vision-language model. In that space an edge between a source patient and a target patient means "these two show the same findings." Each edge can therefore be read and checked by a clinician.
The proposal tests two claims. The first is that propagation over the concept graph yields better pseudo-labels at an unlabeled hospital than propagation over raw embeddings. The second is that one graph statistic, how often a target image's nearest neighbors come from another site, works as a label-free signal for when transfer is succeeding. This second claim addresses a problem the domain-shift literature leaves open: in real deployments there are no target labels for choosing hyperparameters. If concept-space propagation turns out to be no more accurate than raw-space propagation, the proposal still has a falsifiable second claim to stand on. We say in advance how each outcome will be read.
Core Claim
Two pieces of prior work set up the gap. Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Models motivates the concept space. It scores chest radiographs against GPT-4-generated clinical descriptors using the BioViL vision-language model and trains a linear head on those scores. On the age-confounded NIH-age split it reports 50.7% accuracy versus 9.4% for raw BioViL features, which is evidence that concept scores carry less of the confounding signal. However, it processes every image independently, so unlabeled target images play no role. Graph Concept Bottleneck Models does use graphs with concepts, but its graph links concepts within one image, and it does not evaluate across sites. Our graph has patients as nodes and spans hospitals. Neither paper's design gives a way to adapt to a new site, and neither tells a deployer whether adaptation is working.
Concretely, let $z_i$ be the vector of concept scores for image $i$, standardized separately within each site. Per-site standardization is a label-free correction for the fact that vision-language similarity scores drift with acquisition. Let $S$ be the symmetrically normalized k-nearest-neighbor affinity matrix over source and target images, and let $Y$ hold one-hot labels for source images and zeros for target images. Propagated label scores are
$$F = (1-\alpha)(I - \alpha S)^{-1} Y,$$
where $\alpha \in (0,1)$ controls how far labels spread. The diagnostic is the cross-site mixing rate of target image $i$,
$$m_i = \frac{1}{k}\,\big|\{ j \in \mathcal{N}_k(i) : \mathrm{site}(j) \neq \mathrm{site}(i) \}\big|,$$
where $\mathcal{N}_k(i)$ is the set of its $k$ nearest neighbors. A graph whose target nodes link only to other target nodes ($m_i \approx 0$) cannot carry source labels at all. Our hypothesis is that average mixing in the concept graph tracks target accuracy closely enough to rank configurations without target labels.
Research Program
The work has three parts.
The first is building the graph. We will reuse the concept-scoring pipeline of the Robust CBM paper, with BioViL and LLM-elicited radiographic descriptors, and apply it to source and target cohorts. We then build kNN graphs in three spaces: raw BioViL image embeddings, per-site-standardized raw embeddings, and per-site-standardized concept scores. The propagation rule, the value of $k$ and the tuning budget are identical across all three, so the representation is the only thing that changes.
The second is borrowing a diagnostic idea from imbalanced graph learning. Pure Node Selection for Imbalanced Graph Node Classification discards nodes whose neighbors mostly carry other labels, and GraphSHA: Synthesizing Harder Samples for Class-Imbalanced Node Classification deliberately seeks such boundary nodes. Under domain shift, a target node can have impure or uninformative neighbors for two different reasons. It may be a genuine borderline case, or it may sit in a pocket made up only of same-site images. Mixing rate separates the two. A node with high $m_i$ and impure propagated labels is clinically ambiguous. A node with low $m_i$ is isolated by acquisition. We will route only high-mixing nodes into self-training and flag low-mixing ones for review, then measure whether this split predicts which pseudo-labels are wrong.
We are deliberately not adding synthetic node generation in the style of GraphSMOTE: Imbalanced Node Classification on Graphs with Graph Neural Networks. The file names homophily dependence and edge noise across domains as unsolved for such generators, and adding them would confound the main question.
The third part is auditing edges. For a sample of cross-site edges, we will list the concepts that contribute most to the edge's similarity and ask whether those shared concepts are consistent with the labels at both endpoints. This turns "the model transferred a label" into a checkable statement such as "the label came from three source patients sharing effusion and blunted costophrenic angle." It also addresses the bypass worry raised in the interpretability literature, because the propagation never sees raw pixels.
Baselines
The naive reference point is the source-trained concept bottleneck from Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Models, applied unchanged to the target site. Its linear head is fit on source concept scores with the same concept set we use. Any gain from propagation must exceed what the identical concept representation achieves with no use of target data. Its reported reference number is 50.7% on NIH-age. For cross-institution transfer from NIH-CXR to Open-i, no reported result found on Lacuna, so we will measure it ourselves.
The strongest existing method is Noisy Student self-training. This is iterative pseudo-labeling with strong augmentation on a black-box DenseNet, evaluated in Extending the WILDS Benchmark for Unsupervised Adaptation. That paper reports out-of-distribution accuracy rising from 70.8% (ERM) to 86.7% on Camelyon17-WILDS. We will run it with the same unlabeled target pool, the same number of hyperparameter trials, and validation restricted to held-out source sites, since target labels are assumed unavailable. Beating it on radiographs means matching or exceeding its target accuracy while also providing an edge-level explanation. If we exceed it on accuracy alone, that is a strong result. If we trail it by more than a few points, the interpretability argument has to carry the method, and we will report that as such.
The ablation that isolates the key ingredient is the identical propagation run on raw BioViL embeddings, with and without per-site standardization. This decides whether the concept space itself matters or whether per-site normalization and transduction alone explain any gain. A fourth comparison is Graph CBM from Graph Concept Bottleneck Models, trained on source only. It tests whether graphs among concepts help under shift, as opposed to graphs among patients.
Why not just…?
Why not just run Noisy Student on a standard network? On histopathology this is a serious threat. The 70.8% to 86.7% gain on Camelyon17 is large, and our method may not match it there. The same WILDS study, however, reports that self-training's success relies on strong label-preserving augmentations, which are reliable for histology patches but not established for radiographs. It also offers no explanation of why any pseudo-label was assigned. The head-to-head comparisons on chest X-ray and Camelyon17 settle where each approach wins. We will not claim that a loss on Camelyon17 is a win.
Why not just do label propagation on raw foundation-model embeddings? This is the most direct threat, and it may match us on accuracy. It is therefore the main ablation, not a straw man. If raw-space and concept-space propagation are equally accurate, the remaining claims are three. First, concept-space mixing is a better label-free predictor of target accuracy. Second, concept edges are auditable. Third, concept graphs are less tied to confounders, which we test on the NIH-age and NIH-gender splits, where raw BioViL features are known to fail badly.
Why not just align features with CORAL or a domain discriminator? WILDS: A Benchmark of in-the-Wild Distribution Shifts reports that invariance methods such as CORAL and IRM failed to beat plain ERM on 8 of 10 real-world datasets. That makes them weak candidates as the strongest competitor. Our per-site standardization is the cheapest version of moment matching, and the raw-standardized ablation measures exactly how far it gets on its own.
Proposed Experiments and Decisive Checks
The primary setting is chest radiography with NIH-CXR as source and Open-i as target, using the binary normal/abnormal task and concept set of the Robust CBM paper. A secondary setting uses the confounded NIH-age and NIH-gender splits, where we report target accuracy and the fraction of cross-site edges that also cross the confounding group. Camelyon17-WILDS, with three training hospitals and two held-out hospitals, serves as a stress test. The file names no pathology vision-language model, and generic CLIP is documented to transfer poorly to histopathology. We therefore predict that concept-space propagation underperforms there. If it does, that supports the view that gains depend on concept quality. Metrics are target accuracy and balanced accuracy, pseudo-label precision split by mixing rate, and Spearman correlation between graph mixing and target accuracy across a sweep of $k$, $\alpha$ and concept-set size.
Three checks decide the proposal. First, concept-space propagation must beat the source-only concept bottleneck on the target site. If it does not, transduction adds nothing. Second, the concept-space and raw-space propagation runs settle whether the concept representation matters for accuracy, as described above. Third, average mixing rate in the concept graph must rank configurations by true target accuracy, as measured by the Spearman correlation across the sweep. It must also do so better than the same statistic computed on the raw graph, and better than source-site validation accuracy. If mixing rate fails this test, the label-free model-selection claim is withdrawn, and the work reduces to an interpretable but ordinary transductive baseline.
References
- Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Models
- Graph Concept Bottleneck Models
- Pure node selection for imbalanced graph node classification
- GraphSHA: Synthesizing Harder Samples for Class-Imbalanced Node Classification
- GraphSMOTE: Imbalanced Node Classification on Graphs with Graph Neural Networks
- Extending the WILDS Benchmark for Unsupervised Adaptation
- WILDS: A Benchmark of in-the-Wild Distribution Shifts