When speech-enhancement algorithms are scored only by mathematical distance measures like PESQ, they learn to exploit the metric's blind spots: benchmark numbers climb while the audio sounds worse to human listeners. The audio field diagnosed this quantitatively — in the INTERSPEECH 2020 Deep Noise Suppression Challenge, the baseline model registered a near-zero subjective improvement of 0.02 dMOS while top systems reached 0.57–0.67, proving that objective proxies and human perception diverge sharply — and then fixed it with an open dataset and a standardized human-rating protocol The INTERSPEECH 2020 Deep Noise Suppression Challenge. The gut-microbiome field has the same disease with no equivalent diagnosis. Models of antibiotic-driven microbiome disruption are judged almost exclusively by relative-abundance forecasting error, static diversity indices, or correlation coefficients — mathematical proxies that can declare a "recovered" ecosystem while the underlying community remains scarred and vulnerable: broad-spectrum meropenem leaves the mouse gut measurably unrestored 30 days after the infection is cleared Absence of gut microbiota restoration following meropenem treatment in a systemic infection model, and a second antibiotic course in infants produces disruptions that persist for up to four months even after superficial recovery from the first Revealing the temporal dynamics of antibiotic anomalies in the infant gut microbiome with neural jump ODEs.
This proposal treats antibiotic-induced dysbiosis as the application regime, the objective-metrics problem as the core mechanism, and dataset transparency as both an evaluation constraint and a deliverable. The claim is specific: the evaluation layer, not model architecture, is the current bottleneck in antibiotic-dysbiosis modeling, and the field cannot know whether its models work because (a) the standard metrics are blind to exactly the failures that matter clinically, and (b) the preprocessing pipelines behind the headline numbers have never been independently audited. We will quantify both problems on existing public cohorts and release the first transparent, dual-metric benchmark for antibiotic-perturbation microbiome modeling.
Core Claim
Standard objective metrics in this field — aggregate abundance error, cross-sectional diversity, and taxon-to-metabolite correlation — decouple from functionally meaningful recovery in the same way PESQ decouples from perceived speech quality. The mechanism is known: antibiotics selectively destroy minority functional guilds, particularly butyrate-producing Clostridiaceae, whose loss starves colonic epithelial cells, dissolves tight junctions, and lets bacterial endotoxin leak into circulation Effects of antibiotic therapy on the early development of gut microbiota and butyrate-producers in early infants, Gut Microbiota Remodeling Mediates the Therapeutic Effects of a Plant-Based Medicine on DSS-Induced Ulcerative Colitis in Mice via the Butyrate-SVCT1-Vitamin C Axis. A model can therefore achieve excellent aggregate abundance accuracy while misestimating precisely these minority taxa, and a community can score as "recovered" on diversity while being depleted of the guilds that hold the barrier together. Clinical evidence already hints at this decoupling: in acute myeloid leukemia patients, reductions in overall diversity did not uniformly track expansion of resistance genes or resistant colonization Characterization of Antibiotic Administration Factors Associated with Microbiome Disruption and Subsequent Antibiotic-Resistant Infection and Colonization Events in Acute Myeloid Leukemia Patients Receiving Chemotherapy. We claim that this divergence is measurable, large, and currently unquantified — and that fixing it requires the paired treatment the audio field used: functionally grounded metrics reported alongside proxies, on fully transparent data.
Research Program
The program has three workstreams, each reusing an established method from one of the three literatures rather than inventing new machinery.
The first workstream quantifies the proxy gap. On the DIABIMMUNE infant cohort (281 infants, longitudinal 16S profiles with exact antibiotic timing), we define a second-hit vulnerability prediction task: using only data taken after a first antibiotic course, predict the magnitude and duration of disruption following a second course. This is the ground truth on which static metrics provably fail — the neural-jump-ODE study showed that static diversity snapshots display superficial recovery after course one while course-two anomalies persist up to four months Revealing the temporal dynamics of antibiotic anomalies in the infant gut microbiome with neural jump ODEs. We will measure how often each candidate metric — Shannon diversity, observed richness, Faith's phylogenetic diversity, trajectory-based anomaly scores, and a new guild-weighted score — classifies a scarred post-course-one community as "recovered." The guild-weighted score is deliberately simple: held-out prediction error computed separately for butyrate-producer-affiliated taxa and for the aggregate community, reported side by side. The point is not that the weighting is clever; the point is that nobody has measured whether the field's leading models pass or fail it, and whether model rankings flip when it is added. This mirrors the decisive move in speech, where metric GANs that directly optimize PESQ still diverge from human perception CMGAN: Conformer-based Metric GAN for Speech Enhancement and objective measures fail outright on pathological voices Voice enhancement of male speakers with laryngeal neoplasms — the dysbiotic gut is the microbiome field's pathological domain.
The second workstream is a reproducibility audit, transplanting the method that connected metrics to transparency in digital pathology. Fell et al. independently reimplemented three top Camelyon challenge algorithms using only the published text, and performance collapsed — the Lee algorithm fell from 92% to 54.7% slide accuracy, and the Liu algorithm's lesion-detection score fell from 0.981 to 0.499 — because tissue thresholds, stain normalization, and split boundaries were undocumented Reproducibility of deep learning in digital pathology whole slide image analysis. Microbiome multi-omics pipelines are at least as knob-laden: amplicon-variant clustering thresholds, zero-count handling, centered log-ratio transformation of compositional data, metabolite peak alignment, and patient-level versus sample-level splits (a leakage risk the pathology audit flagged directly). We will reimplement the two leading microbiome-metabolome pipelines from their papers alone — the sparse encoder-decoder of Deep in the Bowel: Highly Interpretable Neural Encoder-Decoder Networks Predict Gut Metabolites from Gut Microbiome and the multi-layer perceptron framework of MiMeNet: Exploring microbiome-metabolome relationships using neural networks — and then sweep the undocumented preprocessing choices to measure how far their headline numbers move. We also extend Fell et al.'s 12-point reproducibility checklist into a microbiome-specific version, prioritized by which knobs our sweep shows to matter most. MDSINE2 raises a related caveat we inherit: relative-abundance data without absolute qPCR concentrations can introduce artifacts that obscure true population collapses Learning ecosystem-scale dynamics from microbiome data with MDSINE2, so transformation choices are not cosmetic and belong in any audited pipeline.
The third workstream packages the results as a benchmark in the mold of the DNS Challenge: one harness, documented provenance for every preprocessing decision, enforced infant-level and patient-level splits, and dual reporting of proxy and guild-weighted metrics for any submitted recovery or forecasting model, evaluated across DIABIMMUNE, the Franzosa/PRISM multi-omics cohorts, and the gnotobiotic mouse perturbation data with its unusually dense sampling (76 time points per mouse over 65 days, with dietary and vancomycin/gentamicin antibiotic perturbations) Learning ecosystem-scale dynamics from microbiome data with MDSINE2. The DNS Challenge showed that transparency plus a trusted ground-truth layer, not transparency alone, is what let a fragmented field compare fairly The INTERSPEECH 2020 Deep Noise Suppression Challenge.
Baselines
The naive reference point is the "do nothing" practice the field already uses: cross-sectional diversity metrics — Shannon index, observed richness, and Faith's phylogenetic diversity computed at single time points — used to classify whether an antibiotic disruption has occurred and whether the community has recovered. This baseline is not hypothetical; on DIABIMMUNE it was run head-to-head against dynamical models and achieved a macro-averaged F1 of 0.496 for detecting antibiotic events, reported in Revealing the temporal dynamics of antibiotic anomalies in the infant gut microbiome with neural jump ODEs. It is the strongest competitor for our claim precisely because it is what most clinical studies actually do, and beating it means demonstrating detection of the second-hit scar that static snapshots structurally cannot see. We will run it on identical infant-level splits, and any recovery-classification result from our harness counts as beating it only if it both exceeds 0.496 F1 on event detection and correctly flags a substantial fraction of post-course-one communities that static metrics call recovered but that later suffer four-month course-two disruptions.
The strongest existing method is the neural jump ODE with signature transforms from Revealing the temporal dynamics of antibiotic anomalies in the infant gut microbiome with neural jump ODEs, which reported a macro-averaged F1 of 0.542 on the same cohort and task — the best reported number we found on Lacuna for antibiotic-event prediction in the infant gut. It is the strongest competitor because it already fixes the static-snapshot failure, models continuous-time development, and incorporates covariates that stop natural milestones like weaning from being misread as antibiotic pathology. For the forecasting side on the mouse perturbation data, the strongest existing method is MDSINE2, the Bayesian module-based dynamical model that reportedly outperformed regularized generalized Lotka-Volterra baselines (gLV-L2, gLV-net) on held-out trajectories; no reported numerical forecasting error is given in the file, so we will reproduce those comparisons ourselves. Fairness means reimplementation from the published text under the same data, the same splits, and a matched tuning budget, with reproduction of the published numbers (0.542 F1; MDSINE2's relative advantage over gLV variants) verified before any perturbation. Beating these baselines is defined carefully: our primary claim is about evaluation, not model superiority, so "beating" means showing that model rankings and recovery verdicts change when the guild-weighted metric replaces or joins the aggregate one — if rankings are identical under both metrics, the proxy gap is small and our claim is disconfirmed.
The key ablation isolates the functional-weighting ingredient: identical model predictions (from the reimplemented NJODE and MDSINE2) scored twice, once with uniform abundance error and once with the butyrate-guild error separated out. A second ablation isolates the transparency ingredient: the reimplemented pipelines scored with full preprocessing disclosure versus paper-text-only reconstruction, exactly paralleling Fell et al.'s design. If the Sparse-NED audit reproduces its published Pearson correlation of 0.74 for the top ten metabolites and its 0.866 AUC for predicting antibiotic use from latent features without privileged information, the transparency gap in this subfield is smaller than in pathology; if it collapses the way the Lee algorithm did, the gap is confirmed.
Why not just…?
Why not just use the diversity metrics everyone already has? Because the evidence against them is already on record and quantitative: static diversity baselines reached only 0.496 F1 on antibiotic-event detection and, more damning, display superficial numerical recovery after a first antibiotic course while a second course triggers anomalies persisting up to four months Revealing the temporal dynamics of antibiotic anomalies in the infant gut microbiome with neural jump ODEs. The meropenem cohort shows the same failure at the endpoint: 30 days after clearance, diversity-style aggregate description misses a structurally unrecovered community Absence of gut microbiota restoration following meropenem treatment in a systemic infection model. Our first experiment directly tests whether any static metric survives the second-hit prediction task; if one does, that would be a genuine finding against part of our claim, and we would report it.
Why not just adopt a reporting checklist, like Fell et al.'s 12 points? A checklist enforces disclosure but cannot say which knobs matter or whether the metrics themselves are valid, and Fell et al.'s audit was confined to gigapixel pathology images, not sparse, compositional, longitudinal sequencing data Reproducibility of deep learning in digital pathology whole slide image analysis. The DNS Challenge is instructive here: it paired open data with a new ground-truth evaluation layer, and the pairing — not disclosure alone — is what exposed the proxy failure The INTERSPEECH 2020 Deep Noise Suppression Challenge. Our audit supplies the evidence a checklist needs to be prioritized rather than pro forma: the sweep tells the field which of the twenty-odd preprocessing decisions actually move the headline numbers.
Why not just add butyrate taxa to the metric and publish that? This is the most serious objection, because the guild-weighted score is indeed a simple idea. But a metric proposed without evidence is just another unvalidated proxy — the exact mistake this proposal criticizes. The contribution is the measurement: whether leading models pass or fail the functional axis, whether rankings flip, and whether the divergence between aggregate and guild metrics is large enough to matter, established across three existing cohorts before anything is released. The pathological-voice result is the cautionary precedent: it took a dedicated study to show objective metrics fail in damaged systems, not the observation that they might Voice enhancement of male speakers with laryngeal neoplasms. If our sweep finds that aggregate and guild-weighted errors track each other closely across models and cohorts, the simple idea is wrong and the proposal's central claim is falsified.
Proposed Experiments
Four concrete studies, each tied to a cohort and a number in the file. First, on DIABIMMUNE, the second-hit vulnerability task: train on infant-level splits, predict post-course-two disruption magnitude from post-course-one data, and report the disagreement rate between each metric's "recovered" verdict and the subsequently observed four-month disruption. Second, the audit: reimplement Sparse-NED and MiMeNet from paper text on the Franzosa cohort (N = 220) and PRISM subset (121 IBD patients, 34 controls), then sweep preprocessing and split choices; the targets to reproduce are Sparse-NED's Pearson 0.74 on top-ten metabolites (against 0.46 for linear regression and 0.47 for canonical correlation analysis), its 0.866 AUC for antibiotic use, and MiMeNet's mean Spearman correlation of 0.309 (against 0.108 for MelonnPan) with 366 versus 198 well-predicted metabolites Deep in the Bowel: Highly Interpretable Neural Encoder-Decoder Networks Predict Gut Metabolites from Gut Microbiome, MiMeNet: Exploring microbiome-metabolome relationships using neural networks. Third, on the gnotobiotic mouse data, rerun MDSINE2, gLV-L2, and gLV-net under a guild-weighted evaluation; no reported numerical forecasting errors exist on Lacuna for these comparisons, so we will establish them fresh and test for rank flips. Fourth, on the meropenem cohort, compute whether day-30 communities score as recovered under aggregate abundance distance but not under guild-weighted composition; no reported result found on Lacuna for error values on this cohort, so this is computed anew as a qualitative but decisive check of proxy blindness.
One honest limitation closes the design. No dataset in the corpus pairs longitudinal microbiome data with direct host-barrier measurements such as circulating endotoxin, so the guild-weighted score is a mechanistically motivated surrogate, not measured host ground truth. The DNS Challenge needed its crowdsourced MOS layer before proxy failure could be proven; this field lacks the equivalent, and our benchmark's specification of exactly what such a cohort must contain — absolute concentrations, documented antibiotic regimens, paired barrier markers, patient-level splits — is positioned to make that layer collectible. Until then, the proposal's falsifiable deliverables stand on their own: the measured size of the proxy gap, the first preprocessing-reproducibility audit of microbiome multi-omics, and an open harness where both kinds of error are always reported together.
References
- The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Results
- Absence of gut microbiota restoration following meropenem treatment in a systemic infection model
- Revealing the temporal dynamics of antibiotic anomalies in the infant gut microbiome with neural jump ODEs
- Effects of antibiotic therapy on the early development of gut microbiota and butyrate-producers in early infants
- Gut Microbiota Remodeling Mediates the Therapeutic Effects of a Plant-Based Medicine on DSS-Induced Ulcerative Colitis in Mice via the Butyrate-SVCT1-Vitamin C Axis
- Characterization of Antibiotic Administration Factors Associated with Microbiome Disruption and Subsequent Antibiotic-Resistant Infection and Colonization Events in Acute Myeloid Leukemia Patients Receiving Chemotherapy
- CMGAN: Conformer-based Metric GAN for Speech Enhancement
- Reproducibility of deep learning in digital pathology whole slide image analysis
- Deep in the Bowel: Highly Interpretable Neural Encoder-Decoder Networks Predict Gut Metabolites from Gut Microbiome
- MiMeNet: Exploring microbiome-metabolome relationships using neural networks
- Learning ecosystem-scale dynamics from microbiome data with MDSINE2