Federated Panel Completion: Predicting Off-Panel Pathogens from Distributed Targeted Diagnostics

When a hospital runs a targeted diagnostic panel — a PCR panel or targeted next-generation sequencing (tNGS) assay that tests a fixed list of pathogens — a negative result is ambiguous: the patient may be uninfected, or infected with an organism the panel simply does not cover. Rich hospitals can fall back on metagenomic sequencing (mNGS), which reads all nucleic acid in a sample, but this costs 20 million reads per sample versus roughly 80,000 for a targeted panel, and is out of reach for most regional facilities. The result is a systematic, structural blind spot: each hospital's patient–pathogen test matrix is missing entries by design, with a different mask of missingness at each site, and no one can pool the raw records to fill the gaps because patient-level test results cannot legally or practically leave the institution.

This proposal argues that the gaps themselves are learnable. Patients, symptoms, and pathogens form an interaction matrix of exactly the kind that recommender systems complete every day, and the field has already shown that replacing linear inner products with a learned non-linear scoring function — a multi-layer network that takes patient and pathogen embeddings and predicts affinity — captures conditional dependencies that dot products miss (Neural Collaborative Filtering). What is missing is a version of this machinery that (a) treats "not on the panel" as a third state distinct from "tested negative" (borrowing zero-inflated modeling from biomedical matrix completion), (b) disentangles shared biology from site-specific laboratory bias (borrowing the dual-encoder split of federated collaborative filtering), and (c) trains across hospitals without moving patient records. No paper on record does all three: the federated non-linear recommender Personalized Federated Collaborative Filtering: A Variational AutoEncoder Approach works only on movie ratings, the neural biomedical completion model Neural Collective Matrix Factorization for Integrated Analysis of Heterogeneous Biomedical Data is entirely centralized, and the privacy-preserving surveillance system EpiOracle: Privacy-Preserving Cross-Facility Early Warning for Unknown Epidemics does no learning at all — it matches binary symptom strings by Hamming distance.

Core Claim

A federation of hospitals running different targeted panels can jointly learn a non-linear completion model that ranks likely off-panel pathogens for a given patient — raising the effective diagnostic yield of an 80,000-read targeted assay toward that of a 20-million-read metagenomic assay — without any patient-level record crossing institutional boundaries. The claim rests on a structural observation the e-commerce literature never had: in a hospital, the mask of missingness is known and public (the panel composition is printed on the kit), so we can train explicitly on the difference between "tested and negative" and "never tested," rather than guessing at missingness as movie-ratings models must.

Concretely, for specimen $s$ and pathogen $p$, the model predicts

$$\hat{y}_{sp} = f_\theta\!\big(g(z_s),\, q_p\big),$$

where $z_s$ is a latent representation of the specimen (derived from whatever the site did measure: on-panel results, cycle-time values, host-response markers if available), $g$ and $q_p$ map into a shared pathogen space, and $f_\theta$ is a feedforward network rather than an inner product. The likelihood is zero-inflated in the sense of Neural Collective Matrix Factorization for Integrated Analysis of Heterogeneous Biomedical Data: observed negatives, unobserved (off-panel) entries, and positives are modeled as three distinct data states, not two. Following the dual-encoder design of Personalized Federated Collaborative Filtering: A Variational AutoEncoder Approach, the pathogen embedding table and the scoring network $f_\theta$ are aggregated on a server (they contain no patient information), while each site keeps a private encoder that absorbs its own laboratory's batch effects and assay quirks. This split directly targets the platform-transfer problem documented in Development of Machine Learning Classifiers for Blood-based Diagnosis and Prognosis of Suspected Acute Infections and Sepsis, where models collapsed when moved between microarray and point-of-care instruments: here, the transferable part is trained as transferable from the start, and the site-specific part never moves.

Research Program

The work has three components. First, we build the model itself — federated, zero-inflated, with a non-linear completion head — and specify the three-state observation model, where the mask $M_{sp} \in \{\text{tested}, \text{off-panel}, \text{unmeasured}\}$ is known per site. Second, we exploit an asymmetry no e-commerce federation has: a few sites (academic centers) run mNGS and therefore observe dense rows — nearly the full pathogen column set — while most sites observe only panel-restricted rows. The dense sites act as supervisors for the global pathogen embedding table and the scoring network; the sparse sites contribute specimen representations and local encoders. This mirrors the real cost structure documented in Comparison of the Impact of tNGS with mNGS on Antimicrobial Management in Patients with LRTIs: A Multicenter Retrospective Cohort Study, where both assay types coexist across three hospitals. Third, we run a controlled sparsity sweep to resolve a genuine contradiction in the literature: deep interaction models win on relation prediction (Neural Collaborative Filtering; Neural Collective Matrix Factorization for Integrated Analysis of Heterogeneous Biomedical Data, which reports AUC 0.8596 against 0.8245 for the best linear baseline), yet linear logistic regression beat multi-layer perceptrons under platform transfer in Development of Machine Learning Classifiers for Blood-based Diagnosis and Prognosis of Suspected Acute Infections and Sepsis. Where the crossover sits, as a function of matrix density and cross-site heterogeneity, is an open empirical question worth answering in its own right.

Baselines

The "do nothing" reference point is current practice: accept the panel's answer as final, with escalation to mNGS only for high-risk patients. The reference numbers come from Comparison of the Impact of tNGS with mNGS on Antimicrobial Management in Patients with LRTIs: A Multicenter Retrospective Cohort Study, which reports a pathogen yield of 89.74% for tNGS against 89.71% for mNGS ($P=0.991$) at 80,000 versus 20,000,000 reads per sample. A completion model earns its keep only if, on the specimens where the panel missed and mNGS (had it been run) would have found the agent, the model places the true pathogen near the top of its off-panel ranking — in effect closing part of the residual ~10% miss gap without spending extra reads. A naive reference point sits just above doing nothing: treat every off-panel entry as a negative and fit a classifier, which conflates "not tested" with "not present" and should fail visibly; we include it to show the three-state observation model matters.

The strongest existing competitor is FedDAE from Personalized Federated Collaborative Filtering: A Variational AutoEncoder Approach — the only published federated non-linear collaborative filtering framework, reporting Hit Rate@20 of 0.1232 and NDCG@20 of 0.0503 on MovieLens-100K. We will re-implement it faithfully (validated by reproducing those MovieLens numbers within tolerance on our own hardware before touching clinical data) and run it on our diagnostic matrices with the same federation topology, communication budget, and tuning allowance as our model, but with its two-state (observed/missing) reconstruction objective, since it has no notion of an off-panel state. Beating it means higher off-panel recovery AUROC and higher top-$k$ pathogen recovery at matched communication rounds. The second key comparator is centralized NCMF from Neural Collective Matrix Factorization for Integrated Analysis of Heterogeneous Biomedical Data (AUC 0.8596 on PubMed relation prediction), trained on pooled data as a privacy-violating upper bound: the gap between our federated model and pooled NCMF measures the price of privacy. Finally, the ablation that isolates our key ingredient replaces the non-linear completion head with a plain dot product while keeping everything else fixed; a second ablation removes the site-specific local encoder; a third collapses off-panel entries into negatives to isolate the three-state observation model.

Why not just…?

Why not just use linear matrix factorization or logistic regression? This is a real threat, not a strawman: Development of Machine Learning Classifiers for Blood-based Diagnosis and Prognosis of Suspected Acute Infections and Sepsis found logistic regression outperformed multi-layer perceptrons under platform transfer on 29-mRNA host-response data, precisely the regime we target. The counter-evidence is that this advantage appeared on low-dimensional tabular biomarkers, whereas on relational prediction across heterogeneous matrices the neural completion head of Neural Collective Matrix Factorization for Integrated Analysis of Heterogeneous Biomedical Data beat every linear and graph baseline (AUC 0.8596 versus 0.8245 for ComplEx). We will not argue this; we will measure it. The sparsity sweep runs federated linear matrix factorization and federated logistic regression under identical budgets at every operating point, and if the linear model matches the neural head at realistic panel sizes, we report that as the headline finding — the dual-encoder site-bias separation and the three-state mask would then be the surviving contributions, and the sweep itself would resolve the contradiction documented above.

Why not just pool the data centrally with anonymization? Central pooling is what Neural Collective Matrix Factorization for Integrated Analysis of Heterogeneous Biomedical Data does, and it is the strongest accuracy baseline — but anonymization does not discharge the constraints: raw patient-level infectious-disease test records are restricted by the same privacy regimes that motivated federated deployments in medicine generally (Opportunities and obstacles for deep learning in biology and medicine), and per-sample sequencing data is far too heavy to ship routinely. The centralized run therefore serves as an upper bound, and the experiment quantifies exactly what accuracy federated training sacrifices. If the sacrifice is near zero, the federated design is justified; if it is large, we will say so and investigate which component (pathogen-table aggregation or specimen encoding) pays the cost.

Why not just run mNGS on everyone, or expand the panels? The mNGS option is foreclosed by cost and setting: Comparison of the Impact of tNGS with mNGS on Antimicrobial Management in Patients with LRTIs: A Multicenter Retrospective Cohort Study shows a 250-fold read-depth difference, and metagenomics remains sensitive to host background and dependent on infrastructure most facilities lack. Expanding panels does not escape the underlying trap: a panel is a hypothesis about which pathogens matter, and it fails exactly on the rare, novel, or engineered agents nobody pre-synthesized primers for — the failure mode documented for biothreat agents in Targeted amplification for enhanced detection of biothreat agents by next-generation sequencing and for the hidden psittacosis burden in Epidemiological characteristics of human psittacosis in Guangdong Province, China, 2019–2024. Our model is a software overlay on existing 80,000-read assays: it proposes which unmeasured column to test next, which a wider panel cannot do by construction.

Why not just use EpiOracle's encrypted cross-facility matching? EpiOracle: Privacy-Preserving Cross-Facility Early Warning for Unknown Epidemics is attractive on cost — 12.4 ms facility-side latency and under 40 KB per 100 checks — but it is Hamming-distance matching over binary symptom strings with roughly 70% accuracy, no learned representation, and no mechanism for imputing an unobserved pathogen column. It solves outbreak detection; we solve per-specimen completion. The two are complementary, and we will say so rather than claim to replace it.

Proposed Experiments

The primary testbed is the LRTI multicenter cohort of Comparison of the Impact of tNGS with mNGS on Antimicrobial Management in Patients with LRTIs: A Multicenter Retrospective Cohort Study: 321 hospitalized lower-respiratory-tract-infection patients across three hospitals, with 204 mNGS samples providing dense, near-complete pathogen observations and 117 tNGS samples providing panel-restricted ones. We construct a patient–pathogen matrix whose columns are all pathogens detected anywhere in the cohort (the most frequent being Aspergillus), assign each tNGS site its actual panel as the missingness mask, and ask the federated model to rank off-panel pathogens. The decisive metric is recovery of mNGS-confirmed positives that fall outside a site's panel, measured by AUROC and recall at $k$ (top-3, top-5) — the clinically actionable quantity is "which one or two extra assays should this site order." The mNGS arm doubles as ground truth: we can simulate arbitrarily many panel designs by masking the dense matrix, giving a factorial design over panel breadth (10–200 targets), site heterogeneity, and federation composition (one rich site versus many).

A second experiment tests whether the private local encoder actually absorbs site effects, using the platform-transfer setting of Host-response transcriptional biomarkers accurately discriminate bacterial and viral infections of global relevance (whole-transcriptome discovery AUROC 0.93 degrading to 0.84 on the 263-transcript NanoString validation platform) as a template: we train with one site's measurement process and evaluate on another's, comparing the dual-encoder model against a single shared encoder. The host-response side of the matrix is optional input, not a competing classifier — we are not trying to beat the TriVerity bacterial/viral/none AUROC of 0.83 reported in Development of Machine Learning Classifiers for Blood-based Diagnosis and Prognosis of Suspected Acute Infections and Sepsis; we ask only whether adding host-response columns to the specimen representation improves off-panel pathogen ranking.

Third, we validate the pipeline mechanics on MovieLens-100K against the published FedDAE numbers (HR@20 0.1232, NDCG@20 0.0503) and on the PubMed relation benchmark of Neural Collective Matrix Factorization for Integrated Analysis of Heterogeneous Biomedical Data (NCMF AUC 0.8596, ComplEx 0.8245) under simulated federation — a sanity check that our federated implementation loses little to centralization on data where the centralized numbers are known. No reported result exists on Lacuna for federated biomedical relation completion, which is precisely the gap this proposal fills. The sparsity sweep across all testbeds then maps where the neural head beats the linear one, resolving the linear-versus-deep contradiction in one place with one protocol. Success is defined in advance: the federated model must (i) beat FedDAE re-implemented on the same diagnostic matrices, (ii) recover at least half the centralized-NCMF-to-federated accuracy gap, and (iii) place a mNGS-confirmed off-panel pathogen in the top-5 for a meaningful fraction of missed cases — the point at which an 80,000-read assay plus a model starts to do the work of a 20-million-read one.