The diagnostic signal is in the edges: microbial co-abundance interactions as a missing marker view

When clinicians build a diagnostic test from microbial sequencing data, they almost always start the same way: find the taxa whose abundance differs between patients and healthy people, and build a panel from those. But ecological evidence says much of the disease signal does not live in any single microbe. In a 6,999-person prospective cohort, 40.3% of the genera participating in hypertension-associated co-abundance networks showed no significant change in their own abundance, yet those genera took part in 68.2% of the disease-associated interactions and modulated 84 disease-severity metabolic pathways (Non‐differential gut microbes contribute to hypertension and its severity through co‐abundances: A multi‐regional prospective cohort study). Meanwhile, the multi-marker diagnostics literature has already solved the analogous problem on the host side: pairing a high-sensitivity marker with a high-specificity one via logistic combination lifts ovarian-cancer AUC from 0.853 (CA 125 alone) to 0.952 (ROMA) (HE4 a novel tumour marker for ovarian cancer: comparison with CA 125 and ROMA algorithm in patients with gynaecological diseases). Nobody has applied that combinational logic to the interaction structure of a microbial community.

The proposal in one sentence: turn co-abundance edges — pairwise relationships between microbes, including microbes that individually look healthy — into explicit diagnostic features, and fuse them with host markers through multi-view stacking, so that a diagnostic model can use "community pattern" signal that every abundance-based panel currently throws away.

Core Claim

Diagnostic signal in microbiome data is distributed across pairwise interactions, not concentrated in differentially abundant taxa. Three lines of evidence in the literature motivate this: the hypertension co-abundance result above; the finding that permuting away individual OTUs causes no sharp AUROC drop in colonic neoplasia detection, indicating distributed community patterns rather than single-taxon drivers (A Framework for Effective Application of Machine Learning to Microbiome-Based Classification Problems); and the observation that colorectal carcinogenesis proceeds through shifts of whole multi-species metacommunities, not one enriched pathogen — Metacommunity E (an oral consortium including Fusobacterium, Parvimonas, Gemella, Leptotrichia) appears in 48% of carcinomas and is absent from healthy mucosa dominated by other metacommunities (Gut mucosal microbiome across stages of colorectal carcinogenesis). Bayesian compositional regression independently supports the small-effects side: when 70% of effects are small, a model that captures random-effect covariance achieves simulated $R^2 = 0.814$, far above purely sparse selection (Bayesian compositional generalized linear mixed models for disease prediction using microbiome data).

The claim is falsifiable and operational: a classifier whose microbial view is built from edge features — log-ratios along disease-associated co-abundance pairs — will match or beat the same classifier built from taxon abundances on fixed public benchmarks, and the subset of edges whose both endpoints are non-differential in abundance will alone carry predictive signal. If neither holds, the ecological findings are epidemiologically real but diagnostically useless, and that too would be worth knowing.

Research Program

Three components, in increasing order of risk.

First, an edge-feature constructor that respects compositionality. Microbial sequencing yields relative abundances that sum to one, which induces spurious negative correlations and breaks standard models (Bayesian Graphical Compositional Regression for Microbiome Data). Within training folds only (never across the train/test boundary), we infer a co-abundance network with CCLasso, which corrects for the sum-to-one constraint, as used in the hypertension cohort (Non‐differential gut microbes contribute to hypertension and its severity through co‐abundances: A multi‐regional prospective cohort study). Each selected edge between taxa $i$ and $j$ becomes a feature

$$f_{ij} = \log \frac{x_i + \epsilon}{x_j + \epsilon},$$

where $x_i$ and $x_j$ are the two taxa's relative abundances and $\epsilon$ is a small constant handling zeros. A log-ratio is the natural unit of compositional data: it is invariant to the total count and directly encodes the balance between two community members. We also build two network-level summaries: edge density inside disease-associated modules, and a polarization-style index along the lines of the MCPI used to track dysbiosis from adenoma to carcinoma (Gut mucosal microbiome across stages of colorectal carcinogenesis).

Second, fusion with host markers via multi-view stacked generalization. The closest existing work, Diagnosis of Inflammatory Bowel Disease and Colorectal Cancer through Multi-View Stacked Generalization Applied on Gut Microbiome Data, trains diverse base learners separately on a 16S abundance view and a clinical-metadata view, then trains a meta-learner on their cross-validated probabilities; on pediatric IBD it reached test average precision 0.69 versus 0.61 for the best single base learner (an MLP) and 0.63 for a SoftVote ensemble. We reuse this exact architecture but give it three views: raw abundances, our edge features, and host/clinical markers. This is the crucial contrast with the existing work: their microbial view is an abundance table filtered by variance and ANOVA — a per-taxon differential filter of exactly the kind the hypertension cohort shows discards disease-relevant structure. Our contribution is a new kind of view, not a new ensemble.

Third, an honest test of where edge features stop helping. We will include frozen embeddings from a pretrained microbiome language model (Waypoint, trained on the 539,308-sample Atlas corpus) as an optional fourth view, because the literature leaves open whether foundation-model representations capture the same community signal our edges capture (LLMs Can Learn the Language of the Microbiome). That paper reports a hard threshold — pretrained transformers only reliably beat shallow baselines once fine-tuning cohorts reach roughly 10,000 labeled samples, and show negative scaling below that — so we use Waypoint strictly as a frozen feature extractor, never fine-tuned, which respects the small cohorts we work with ($N = 490$ and below).

One limitation we state up front: the file contains no public dataset pairing stool microbiome sequencing with rich plasma host-marker panels (proteins, cfDNA fragmentomics). The multi-marker fusion is therefore evaluated on the paired 16S-plus-clinical-metadata IBD dataset, and the program includes assembling a prospective paired sub-cohort (stool 16S plus a small plasma protein panel) to test the full three-modality version; until that exists, the fusion claim is limited to clinical covariates.

Proposed Experiments

Benchmark 1, the primary test: the colonic screen-relevant neoplasia dataset — fecal 16S profiles from 490 individuals (229 neoplasia cases, 261 controls), evaluated with 100 repeated 80/20 splits and five-fold cross-validation inside each training split, exactly the protocol of A Framework for Effective Application of Machine Learning to Microbiome-Based Classification Problems. That paper reports median test AUROC of 0.680 for L2-regularized logistic regression (12 minutes training) and 0.695 for Random Forest (83.2 hours). We report median AUROC plus sensitivity at 95% specificity, the operationally relevant metric for screening. Decisive sub-test: an edge-only model using only edges whose both endpoints are non-differential in abundance; if its AUROC exceeds ~0.6, the core claim survives.

Benchmark 2, the multi-marker test: the pediatric IBD dataset with 16S plus clinical metadata under the 80/20 split of the stacked-generalization paper, metric average precision, against their reported 0.69 for the stacked abundance-plus-clinical model. The sibling CRC dataset from the same paper has no reported result found on Lacuna for the stacked classifier's average precision (the text truncates), so we treat it as a held-out replication, not a baseline comparison.

Benchmark 3, the Bayesian frontier: the American Gut Project IBD data, where BCGLMM reaches cross-validated AUC 0.702 against 0.687 for its sparse-only variant and 0.672 for standard GLM (Bayesian compositional generalized linear mixed models for disease prediction using microbiome data). Beating 0.702 with a method that trains in minutes rather than MCMC hours would show edge features capture what mixed modeling captures, at far lower cost — addressing the computational-scalability open problem that paper itself flags.

Benchmark 4, the epidemiology-to-diagnostic translation: the Guangdong Gut Microbiome Project hypertension cohort, asking whether edge features predict hypertension and severity tier where the original study stopped at association and pathway-variance regression. Benchmark 5, stage structure: the carcinogenesis mucosal cohort, testing whether edge polarization separates healthy, adenoma, and carcinoma samples — the original study's cross-sectional design leaves this as a descriptive, not predictive, question.

Baselines

The naive reference point is a single differentially abundant taxon, the microbial analogue of CA 125 alone: the top fold-change genus used as a univariate classifier, and, as its clinically real version, a Fusobacterium-style targeted qPCR. The literature predicts it fails: permuting away individual OTUs causes no sharp performance drop in neoplasia detection (A Framework for Effective Application of Machine Learning to Microbiome-Based Classification Problems), and single-marker tests leave large sensitivity gaps elsewhere — CA 125 alone reaches only 40.9% sensitivity in stage IV endometriosis (The coagulation status in women of endometriosis with stage IV). Beating it is necessary but nearly meaningless; it calibrates the floor.

The strongest existing competitor is multi-view stacked generalization on the abundance-plus-clinical views (Diagnosis of Inflammatory Bowel Disease and Colorectal Cancer through Multi-View Stacked Generalization Applied on Gut Microbiome Data) — strongest because it is the only published system that already combines a microbial view with host information, reporting test average precision 0.69 on pediatric IBD versus 0.61 for the best single base learner and 0.63 for SoftVote. On the SRN benchmark the strongest single-model competitor is Random Forest at median AUROC 0.695, with L2 logistic regression at 0.680 as the efficiency reference. We run all of these under identical protocols (same splits, same 100-repetition scheme, same hyperparameter search budget measured in candidate configurations, same CPU-hours cap) so that any difference is attributable to the feature view, not tuning. Success is defined in advance: median AUROC above the Random Forest figure on SRN with training cost closer to the 12-minute logistic regime than the 83-hour Random Forest regime, and average precision above 0.69 on the IBD multi-view benchmark for the three-view stack.

The key ablation isolates the proposal's one new ingredient: the identical stacked architecture with the edge view deleted, leaving abundance-plus-clinical only. That ablation is the closest prior method, so the experiment doubles as the head-to-head. A second ablation splits the edge view into differential-endpoint and non-differential-endpoint edges; the non-differential block is the component no existing pipeline could produce, and its standalone performance is the cleanest evidence for or against the core claim. A third ablation adds frozen Waypoint embeddings as an extra view to test whether pretrained sequence representations already subsume edge information.

Why not just…?

Why not just use the existing multi-view stacking on abundances and clinical data? This is the real threat, and the proposal is built so the question is settled empirically rather than argued: the abundance-plus-clinical ablation is the closest prior method run under identical conditions. If the edge view adds nothing to it, the simpler method stands and this proposal's diagnostic claim fails — though the non-differential-edge sub-test would still tell us whether the failure is because the hypertension finding does not generalize to colorectal and IBD cohorts. The ecological evidence says the simpler method is leaving signal on the table (40.3% of network genera invisible to its ANOVA filter), but that evidence is epidemiological, not diagnostic, and this is precisely the gap.

Why not just fine-tune a foundation model like Waypoint or Exai-1? The evidence in the file rules out fine-tuning at our scale: Waypoint shows negative scaling below roughly 10,000 labeled samples, with shallow models winning under 1,000 (LLMs Can Learn the Language of the Microbiome), and our cohorts are 490 and fewer. We therefore test the defensible version — frozen embeddings as a feature view — inside the same stack. If frozen Waypoint embeddings beat our engineered edges, the honest conclusion is that pretraining already captures community structure and the edge constructor is superseded; the ablation is powered to detect exactly that.

Why not just concatenate everything and run Random Forest? On the SRN benchmark, Random Forest at 0.695 barely beats L2 logistic regression at 0.680 while costing 83.2 hours versus 12 minutes (A Framework for Effective Application of Machine Learning to Microbiome-Based Classification Problems), so capacity is not the bottleneck — representation is. A tree ensemble can only use edges if they exist as columns, which is what the edge constructor provides; and the Bayesian evidence says the missing signal is cumulative small effects that sparse splitting handles poorly (sparse BCGLM 0.687 versus mixed BCGLMM 0.702 on real IBD data). That said, we will run Random Forest on the augmented abundance-plus-edge table as a fair rival, because if it absorbs the edge signal as well as the stack does, the stacking machinery is unnecessary and the contribution reduces to the feature constructor — a smaller but still real result.

Decisive Checks

Two pre-registered outcomes decide the proposal. The first is the non-differential edge block on SRN: if edges between abundance-unchanged taxa alone reach AUROC around or above the 0.680 logistic baseline, the GGMP finding has been translated from epidemiology into diagnosis for the first time. The second is the three-view-versus-two-view stack on the IBD benchmark against the published 0.69: any gain must exceed the split-to-split variation across the repeated protocol. A negative result on both, with the frozen-Waypoint ablation also negative, would cleanly establish that community-interaction signal in these cohorts is too weak, too cohort-specific, or already captured by existing representations — a finding that would redirect this intersection toward the prospective paired-cohort design, where the multi-marker fusion can finally be tested on real host panels rather than clinical covariates.