Scoring Medical Image Models at a New Hospital Without Labels, Using the Secret-Key Rate Between Images and Blood Tests

A diagnostic image model that looks excellent on a research benchmark often breaks when a hospital deploys it. On the G1020 fundus set, a glaucoma classifier fell to chance-level AUC. Prospective lung-ultrasound accuracy dropped from 87.8% to 65%. The underlying problem is that the new site rarely has the diagnostic labels needed to find out which model survived the move, and waiting for those labels is how failures reach patients. Many hospitals do have something else for the same patients: routine blood tests drawn alongside the scan. Blood counts do not care about parents' hands in a pediatric X-ray or a mis-angled ultrasound probe, yet they are correlated with the disease. Our proposal is to score candidate image models at the new site, without labels, by how much image-derived information they share with the blood tests that cannot be explained by acquisition nuisance such as device, positioning, or artifacts.

The scoring rule comes from information-theoretic secret key agreement. In that setting, two parties hold correlated observations and want to agree on a key that an eavesdropper, who holds her own correlated side information, cannot learn. In our mapping, "Alice" is the image model's output, "Bob" is the blood panel, and "Eve" is a nuisance record of how the image was acquired. The achievable key rate measures exactly the quantity we want: shared image–lab information that Eve cannot reconstruct. The core claim is that a finite-sample estimate of this rate, computed on unlabeled image–lab pairs from the deployment site, ranks candidate models by their true target-site diagnostic performance better than source-validation accuracy or plain image–lab agreement. The work is therefore a label-free model selection and auditing tool, grounded in a key-rate bound, rather than yet another fusion architecture.

Core Claim

Let $f$ be a candidate image encoder whose output is quantized to a short discrete code $U=f(X)$. Let $Y$ be a discretized blood panel and $Z$ a discrete nuisance descriptor of acquisition. A classical lower bound on the one-way secret key rate says that Alice can extract at least

$$ S(f) \;=\; I(U;Y) - I(U;Z) $$

bits per sample of key that Bob can decode and Eve cannot. Here $I(\cdot;\cdot)$ is mutual information, the average number of bits one variable reveals about another. We also compute a conditional variant, $I(U;Y\mid Z)$, which is the natural upper-side counterpart. Where the alphabets are small enough, we compute the unique information $\mathrm{UI}(Y;U\mid Z)$ as well: the advantage that observing the image code gives about the labs over observing only the nuisance record. Unique Information and Secret Key Decompositions shows this quantity sits between the one-way key capacity and one of the tightest known two-way upper bounds.

The prediction is that, across a pool of encoders trained on curated source data, the rank of $S(f)$ on unlabeled target pairs correlates with each encoder's target-site AUC or F1. It should correlate more strongly than the rank of source validation accuracy, and more strongly than the rank of the agreement term $I(U;Y)$ alone.

Quantizing $U$ to a few bits is deliberate. It is what makes these information quantities estimable on clinical sample sizes. It also sidesteps the computational barrier that the research file flags for unique information in continuous high dimensions.

Why This Pairing, and What Prior Work Does and Does Not Do

Two clinical studies motivate the proposal by showing that blood tests anchor diagnosis when images degrade.

Accurate and intelligent diagnosis of pediatric pneumonia using X-ray images and blood testing data reports that models trained on the curated GZCMC chest X-rays fell from 86% to 50% accuracy on raw clinical images containing hands and positioning artifacts. It then fuses X-rays with four blood markers (AMPNet) to reach an average F1 of 0.7781 on the Hainan cohort.

COVID-BLUeS - A Prospective Study on the Value of AI in Lung Ultrasound Analysis shows that blood markers alone reached 84% accuracy and late fusion reached 83%.

Both papers use labs as extra inputs to a supervised model. Neither uses the image–lab correlation as a label-free signal about whether the image model itself is trustworthy at a new site. Neither separates the part of that correlation carried by acquisition nuisance.

The secret-key literature supplies the scoring principle but has never touched learned representations. Randomness and Dependencies Extraction via Polarization, With Applications to Slepian-Wolf Coding and Secrecy and Efficient One-Way Secret-Key Agreement and Private Channel Coding via Polarization give efficient protocols achieving key capacity for known discrete memoryless sources. Communication for Generating Correlation: A Unifying Survey frames extracting keys from correlation as the dual of generating correlation from keys. We borrow the rate expression and the unique-information bounds. We do not borrow the coding constructions, which assume a known stationary joint distribution and asymptotically long blocks. Our distributions are unknown and our samples are finite.

The closest machine-learning work is How Does Distribution Matching Help Domain Generalization: An Information-theoretic Analysis. It derives information-theoretic generalization bounds and trains with gradient and per-sample representation matching (IDM/PDM), reaching 71.2% average on DomainBed and 70.6% on Colored MNIST versus 14% for ordinary training. It differs from our proposal in three ways:

  • It treats shift as a domain index to align across, not as an eavesdropper whose information is subtracted.
  • It uses no second modality.
  • It is a training method evaluated with target labels, not a label-free selector.

Its impossibility result also constrains our design: distribution distances are unreliable when feature dimension exceeds batch size. That result is our reason to score short quantized codes rather than full embeddings.

Research Program

The first component is a pool of candidate encoders for each clinical task, deliberately spanning known failure modes. The pool includes:

A pool of roughly forty encoders per task gives a meaningful spread in target performance. Each encoder's penultimate features are quantized to a code of 2 to 5 bits, using k-means on source features so that no target labels are touched.

The second component is the estimator. $Y$ is formed by discretizing each blood marker at clinical reference thresholds, which yields a small alphabet. $Z$ is a nuisance descriptor with a separate definition per task:

  • For the pediatric X-rays, $Z$ is annotated presence of hands and positioning category, labeled by non-clinicians for label-free cost.
  • For lung ultrasound, $Z$ is the thoracic zone of the standardized protocol plus a frame-quality bin.
  • For the synthetic testbed, $Z$ is the planted cue.

Mutual informations are estimated with plug-in estimators plus bias correction, and bootstrap intervals are computed over patients. Unique information is computed by the standard constrained optimization over joint distributions with fixed pairwise marginals. That optimization is tractable at these alphabet sizes.

The third component is the analysis of when the score must fail, which we treat as part of the contribution. The subtraction $I(U;Z)$ penalizes any feature correlated with nuisance. Deep learning from “passive feeding” to “selective eating” of real-world data documents that sicker patients produce poorer images, so nuisance and disease are themselves correlated. In that regime the lower bound $S(f)$ over-penalizes genuine disease signal. The conditional and unique-information variants should then be preferred, because they measure what the image adds about the labs beyond what nuisance already reveals. We will state and test this ordering: all three scores agree when $Z$ is independent of disease, and only the conditional scores stay rank-faithful as that dependence grows.

The final component tests whether the score works as a training objective, not only a selector. At the target site, a small encoder head is fine-tuned on unlabeled pairs to maximize a differentiable surrogate of $S(f)$: an image–lab contrastive term minus a nuisance-prediction term. The result is compared against the selection-only use. This is a secondary aim, because it is the more crowded direction.

Baselines

The naive reference point is source-validation selection. The practitioner picks the encoder with the best held-out accuracy on curated source data and deploys it. This is what produced the 86%-to-50% and 87.8%-to-65% collapses reported by Yao et al. and Wiedemann et al. It will be run on exactly the same candidate pool. Beating it means a significantly higher rank correlation, by bootstrap over patients, between score and target AUC/F1, and a higher target metric for the top-selected encoder.

The key ablation is the agreement-only score $I(U;Y)$, with no nuisance term. It is what an image–lab contrastive criterion would reward, and it isolates whether treating acquisition nuisance as an eavesdropper matters. It uses the same codes, estimator, and bootstrap. The proposal wins only if $S(f)$ or its conditional variant beats it on the shortcut-prone portion of the pool, where the shortcut features may share information with the labs through severity.

A second label-free competitor is the prediction-confidence proxy: mean maximum softmax probability or prediction entropy on target images. It is the obvious zero-cost score. There is no reported result on Lacuna for label-free model selection on these clinical cohorts, so we will run all selectors ourselves under identical conditions.

For the secondary training aim, the strongest existing methods are AMPNet-style supervised fusion on Hainan, with a reported average F1 of 0.7781 in the pediatric pneumonia paper, and on COVID-BLUeS the late-fusion model at 83% and the blood-only random forest at 84%, both reported in the COVID-BLUeS paper. IDM from Dong et al. is included as the strongest domain-generalization training method, with the same backbone, compute, and a matched hyperparameter search budget. Our label-free adapted image model is not expected to beat supervised fusion outright. The fair bar is to close a substantial fraction of the gap between source-only image models and supervised fusion without any target labels.

Why not just…?

Why not just use the blood tests alone, since they reached 84% on COVID-BLUeS? This is the most serious threat, and we take it at face value. If labs alone suffice for a disease, an image model adds little, and our score should say so. The unique-information term $\mathrm{UI}(D;U\mid Y)$, computed on labeled validation subsets, directly measures whether the image carries diagnostic information the labs lack. We will report it per task. On Hainan the file reports a 13.9% viral-pneumonia F1 gain from adding images, which suggests the image carries unique signal there. The method's value is in auditing the image model, which is still needed for image-only findings and for sites where labs are drawn late. Using labs at audit time does not presuppose that labs replace images.

Why not just filter out bad images first, as DLIFS does? Filtering reaches 0.994 to 0.997 AUC for quality detection. But Li et al. note that it discards scans that are enriched for disease. Hu et al., in Assessing the Performance of Models from the 2022 RSNA Cervical Spine Fracture Detection Competition at a Level I Trauma Center, show specificity falling to 72.1% on high-quality contrast CT, where the failure is a misleading feature rather than a degraded image. Filtering is complementary: we will compute our score on both the filtered and unfiltered target sets to show whether it still ranks models correctly after gatekeeping.

Why not just maximize image–lab agreement with contrastive learning and skip the eavesdropper? That is exactly the agreement-only ablation above. If it matches $S(f)$ on the shortcut-prone encoders, the security framing adds nothing and we will report that. Our reason to expect a gap is that the planted and real nuisances (hands, probe zone) can correlate with severity and hence with labs. An encoder that reads the nuisance then looks falsely aligned with the labs. Only the eavesdropper term exposes this.

Proposed Experiments

The decisive first check is a controlled semi-synthetic testbed where the answer is known. Colored-MNIST-style images serve as $X$, a noisy second view generated from the digit serves as the "lab" $Y$, and color is a planted cue $Z$ whose correlation with the label is varied from zero to strong. Encoders range from color-reading to shape-reading. The score must track test accuracy under color reversal. Separately, we verify the predicted crossover as $Z$ becomes disease-correlated: $S(f)$ should degrade while the conditional and unique-information variants remain rank-faithful.

The main clinical experiment uses GZCMC as curated source and Hainan as target, with X-ray codes, four binned blood markers, and hand/positioning annotations as nuisance. The metric is Spearman correlation between each selector's score and target F1 across the pool, plus the target F1 of each selector's top pick. We report these against the AMPNet reference of 0.7781.

The second clinical experiment uses public scraped lung-ultrasound data as source and COVID-BLUeS as target, with the reported 65%, 83%, and 84% as reference points. COVID-BLUeS has only 63 patients, so it serves as a stress test of estimator variance rather than a definitive ranking. We will report bootstrap intervals and treat overlapping intervals as a null result.

A falsification check reruns the pediatric experiment with $Z$ replaced by a random, uninformative descriptor. Any advantage of $S(f)$ over agreement-only should disappear. If it does not, the gain is an estimator artifact rather than nuisance removal.

The proposal succeeds if the key-rate score beats both source-validation and agreement-only selection on the synthetic testbed and on Hainan, with the predicted failure pattern appearing where nuisance correlates with disease. It fails informatively if agreement-only does equally well. That result would show that clinical shortcut features rarely share information with blood tests, which is itself useful to know.