Label-Blind Fusion: Unconditionally Private Training of Imaging–EHR Models Across Institutions

A CT scan by itself is a weak predictor of pulmonary embolism. Paired with the patient's health record it becomes a strong one. On RadFusion, an imaging-only model reaches 68.9% accuracy, a records-only model 83.7%, and their simple combination 89.0% (RadFusion). In practice the two halves often live in different places. An imaging center holds the scans. A hospital holds the electronic health record (EHR) and the confirmed diagnosis, which is the training label. Every known way of training one model across that divide leaks something to someone, or rests its safety on an unproven hardness assumption. That matters for records that must stay confidential for decades.

We propose a split that makes the problem tractable with information-theoretic security, meaning security that holds even against an adversary with unlimited computing power. Each institution first turns its own data into a short vector using an encoder trained without any labels. All label-dependent training happens inside a secret-sharing protocol, in which every number is split into random pieces held by three servers, so that no single server learns anything at all. That includes the small fusion model that combines the two vectors, and that learns which CT regions matter given the patient's record. The trained model also stays split, so it never leaks the labels it was fit on. The question we answer empirically is how much of the accuracy of joint fusion survives under this restriction. A second question is whether a fusion model that interacts the two modalities, built so it is cheap to evaluate in secret-shared form, earns its cost over the obvious alternative of averaging two separate predictions.

Core Claim

In the setting where one site holds the images and another holds the records, the expensive deep computation does not need protection. Only label-dependent computation does. If both encoders are trained without labels, the part that touches labels can be confined to a small model of low algebraic degree. A model is low-degree when it uses only additions and a few multiplications, which is exactly what secret-sharing protocols compute cheaply and exactly. Training and inference for that small model can then run with zero information leakage to any single server, apart from each patient's final prediction.

Our specific claim has three parts:

  1. A degree-2 or degree-3 fusion head recovers most of the gap between label-free plaintext probes and a fully joint plaintext model.
  2. In this head, the EHR vector produces weights over a handful of regional CT embeddings and also gates the image features.
  3. This is a polynomial analogue of the gating in JI-ADF and of attention pooling across CT slices.
  4. The same head clearly beats secure late averaging and a secure linear probe on concatenated features.
  5. Its communication cost stays within a small constant factor of secure logistic regression.

What is new is not a new secret-sharing primitive. It is a partition of a multimodal clinical model that puts every label-touching operation inside an unconditionally secure computation. The second new element is a fusion architecture designed to be polynomial, rather than approximated after the fact.

Positioning Against Prior Work

The closest prior work is FedV. FedV trains linear models when features are split across parties by column and one party holds labels. It relies on functional encryption, which is a computational assumption, and on a semi-trusted key authority. It was evaluated only on tabular and digit data. We keep its vertical setting but remove both the hardness assumption and the key authority. We also move from linear models to a cross-modal head operating over imaging embeddings.

COPML supplies our method. It uses Shamir secret sharing and Lagrange coding to train logistic regression with information-theoretic privacy, replacing the sigmoid with polynomials of degree 1 or 3. On CIFAR-10 it reached about 80.45% accuracy, "nearly identical" to plaintext. It also constrains us: it assumes rows are split across parties rather than columns, and it warns that polynomial approximation errors compound in deep networks. We respect that warning by never putting a deep network inside the protocol. Only a shallow, natively polynomial head goes there.

The private aggregation work using nearest-neighbor mixing (Private Aggregation for Byzantine-Resilient Heterogeneous FL) and the one-shot distillation work (Federated One-Shot Learning with Data Privacy and Objective-Hiding) both reach unconditional privacy. However, both assume row-split data. The second also needs a public unlabeled dataset that matches the target domain, and paired public CT-plus-EHR data rarely exists. Our design needs no public data.

Two clinical papers motivate the architecture:

  • Frozen encoders lose accuracy, and mean pooling hides small findings. On INSPECT, frozen foundation encoders reach AUROC 0.680–0.684 against 0.721 for the task-specific CT-LRCN. The authors attribute part of the gap to mean-pooling slices, which dilutes small emboli (Fairness Audit of Medical Imaging Foundation Models). This is why our head learns its regional weighting from labels inside the protocol, instead of pooling locally.
  • Static fusion ignores the weaker modality. JI-ADF shows that static fusion lets the visual stream drown out metadata, and that input-dependent gating fixes this. We reuse that idea in polynomial form.

The same fairness audit also raises a risk we must measure. Its debiasing worked on plaintext representations. Fixed-point quantization and polynomial activations might hurt rare age groups more than common ones, much as reverse-waterfilling sanitization distorts outliers in Utility-Privacy Tradeoffs in Databases.

Research Program

Threat model. There are three computing parties: the imaging center, the hospital, and a third helper, such as a regulator or a cloud enclave run by a different operator. They are semi-honest, meaning they follow the protocol but try to learn from what they see, and at most one is corrupted. Shamir sharing with threshold one gives perfect secrecy against any single party.

We state two limits plainly:

  • Unconditional security of the whole system also requires unconditionally secure channels between the parties. In practice that means pre-shared one-time-pad keys or quantum key distribution of the kind simulated in Brazaola-Vicario et al.. Otherwise the channel layer is computational, and we report results under both assumptions.
  • Patient matching across the two sites is assumed to use a shared institutional identifier, which is out of scope here.

Stage one: label-free local encoders, in plaintext.

  • The imaging center trains or adopts a CT encoder with no access to labels. Candidates are self-supervised pretraining on its own scans or a frozen foundation encoder from the INSPECT audit.
  • Each study becomes $k$ regional embeddings, one per contiguous slab of slices, rather than a single mean-pooled vector.
  • The hospital encodes the EHR window (demographics, vitals, medications, diagnosis codes, labs) into a fixed vector, using a label-free autoencoder or standard feature engineering.
  • Nothing label-dependent happens in this stage. Running it in plaintext therefore leaks no label information by construction.

Stage two: the secret-shared fusion head.

  1. Embeddings are reduced in dimension and quantized to fixed-point values over a prime field, then secret-shared among the three parties.
  2. Labels are secret-shared by the hospital.
  3. The head computes $z = \sum_{j=1}^{k} (q^\top s_j)^2 \, W s_j + g \odot (U s) + V e$. Here:
  4. $e$ is the EHR embedding and $s_j$ is the $j$-th regional CT embedding;
  5. $s$ is the unweighted mean of the regional embeddings;
  6. $q = Ae$ is a query vector computed from the record;
  7. $g = Be$ is a feature gate computed from the record;
  8. $\odot$ is element-wise product;
  9. $A, B, U, V, W$ are learned matrices.
  10. The squared score acts as an unnormalized, nonnegative attention weight. Dropping the softmax normalization is what makes it cheap: division is costly in secret-shared arithmetic, while squaring is one multiplication.
  11. The output goes through COPML's degree-3 sigmoid approximation, and training uses the corresponding polynomial gradient.

The forward pass has multiplicative depth 3–4. Each multiplication costs one round of communication among the three parties, so communication grows linearly in batch size and in $k$.

Inference. The trained weights are never reconstructed. At inference the two sites share a new patient's embeddings, and only the final risk score is opened, to the hospital alone.

What leaks, and what does not. Under the threat model, any single party's view is statistically independent of the labels, of the other party's raw data, of the other party's embeddings, and of the trained weights. The only exception is the opened prediction. The proof is standard composition of Shamir-based protocols, and we will write it out for this specific circuit.

A deployment model of this kind has one structural limit: the encoders never adapt to the task. Our experiments are built to measure exactly how much that costs.

Baselines

Naive reference points. The do-nothing reference is the hospital's EHR-only model trained in plaintext, since the hospital already holds both records and labels. The fusion system must beat it, or collaboration has no purpose. On RadFusion, the reported EHR-only accuracy is 83.7% (RadFusion). We also report the imaging-only probe.

Plaintext ceiling. The strongest existing method is a plaintext joint model trained with labels at a single site. Two versions represent it:

  • End-to-end imaging reference. The reported task-specific number on INSPECT is CT-LRCN at AUROC 0.721, against 0.680–0.684 for frozen foundation encoders (Fairness Audit).
  • Multimodal fusion reference. On RadFusion, late fusion of PENet with ElasticNet reaches 89.0% accuracy. No reported multimodal fusion AUROC on INSPECT was found on Lacuna, so we will establish that number ourselves.

We retrain these plaintext ceilings on our splits, with the same tuning budget as the proposal. The budget is a fixed number of hyperparameter trials selected on the validation split. These models are an upper bound, not a competitor to beat.

Best privacy-compatible competitors. These are the baselines the proposal must actually beat:

  • Secure late averaging. Each modality gets its own logistic head over the same label-free embeddings. Both heads are trained inside the same secret-sharing protocol and their probabilities are averaged, which is the RadFusion recipe made private.
  • Secure linear probe. A logistic head over the concatenated mean-pooled embeddings, essentially COPML adapted to column-split data.

Both use identical encoders, quantization, field size, and tuning budget as the proposal. Beating them means a statistically significant AUROC gain on the INSPECT diagnostic task and on at least half of its prognostic tasks, using paired bootstrap on the test split. As a cost reference, we also run FedV-style functional encryption on the linear probe and compare its runtime and traffic.

Ablations isolating the key ingredient. We run the head without the regional attention term and without the record-driven gate. If the regional term is the active ingredient, its removal should erase most of the gain on the diagnostic task. It should matter less for mortality and readmission, which are driven more by the record than by focal image findings.

Precision ablation. The same head is also trained in plaintext floating point. This separates loss caused by the architecture from loss caused by quantization and polynomial approximation.

Why not just…?

Why not just give the imaging site the labels and use secure late fusion? This is the most serious threat. If diagnoses could be shared, the imaging center would train a task-specific CT model and only two scalar predictions would need secure averaging, which is nearly free. The RadFusion result suggests this is strong.

Our answer is twofold. First, the diagnosis is itself protected health information, and moving it to another institution is precisely the leak this setting forbids. Second, we run this privileged configuration explicitly as a non-private reference. If its advantage over our head is small, the case for keeping labels inside the protocol rests on the security argument alone, and we will say so. If its advantage is large, the cost of label-free encoders is the main open problem, and we will report it as such.

Why not just share labels with label differential privacy or DP-SGD? Differential privacy bounds how much any single record can change the output. That is a weaker and different guarantee than zero leakage. The utility-privacy analysis in Utility-Privacy Tradeoffs in Databases notes that differential privacy can be costly in utility, and the INSPECT audit shows that young patients already sit near chance at AUROC 0.508. We will train the imaging model with randomized-response labels at several privacy levels and compare its accuracy and age-group underdiagnosis gap against our head. If label-DP at a moderate privacy level matches our head, we will report that as a real practical alternative.

Why not just use computational cryptography like FedV or homomorphic encryption? Computational schemes rest on hardness assumptions that may not hold against future algorithms or quantum computers (Rothe). That is a concrete concern for medical records that must stay confidential for decades. FedV also needs a semi-trusted key authority. The honest counterpoint is that our scheme needs an honest majority among three parties, which is a different trust assumption rather than a strictly weaker one. We make that trade explicit and measure the runtime and traffic of both approaches on the same head.

Why not just use frozen foundation embeddings with a plain probe? The INSPECT audit already shows this loses about 0.04 AUROC to a task-specific model. Our secure linear probe baseline is exactly this configuration, run under secrecy. The proposal only matters if the regional, record-gated head closes part of that gap.

Proposed Experiments

Primary benchmark: INSPECT. INSPECT has 23,248 CT pulmonary angiography studies from 19,402 patients, with linked EHR, covering one diagnostic task (PE) and seven prognostic tasks. We simulate the institutional split by giving CT to one party and EHR plus labels to another.

Small-cohort check: RadFusion. RadFusion has 1,837 studies with an 80/10/10 split. It tests behavior at small cohort size and allows direct comparison with its reported unimodal and late-fusion figures.

Metrics:

  • AUROC, and accuracy for RadFusion.
  • Expected calibration error, since polynomial sigmoids may distort probabilities.
  • Age, sex, and race gaps in underdiagnosis rate and true-positive rate, using the metric definitions of the INSPECT audit and of RadFusion.
  • Wall-clock training time, total bytes exchanged, and communication rounds, at batch sizes and region counts $k \in \{1, 4, 8, 16\}$.

Decisive checks. Three outcomes would decide the proposal:

  1. Accuracy. Does the secure head beat secure late averaging and the secure linear probe on PE AUROC? Does it recover at least half of the gap between the secure linear probe and the plaintext end-to-end reference?
  2. Precision loss. Does the quantized, polynomial version stay within 0.01 AUROC of its own floating-point counterpart? Such a result would extend COPML's "nearly identical" finding from logistic regression to a cross-modal head.
  3. Fairness under secrecy. Does the age underdiagnosis gap widen under secret-shared training relative to the plaintext head? If it does, we test whether reweighting groups inside the protocol restores it. This reweighting stays polynomial and therefore cheap. It probes the fairness risk raised by combining the INSPECT audit with reverse-waterfilling.

A negative result on the first check would still be informative. It would show that, in the column-split clinical setting, zero-leakage training is limited mainly by label-free encoders rather than by cryptographic cost, which would redirect effort toward secure adaptation of the encoders themselves.