Charting the Resistome Literature with Grounded Dual Review

When nearly 80% of Acinetobacter baumannii isolates in a national surveillance network are carbapenem-resistant, clinicians fall back on last-resort drugs like colistin, as documented for Chinese surveillance networks by Hui Zhang et al. — and the evidence needed to spend those drugs wisely sits scattered across thousands of molecular epidemiology papers. Which resistance gene rides on which plasmid, in which bacterial sequence type, in which hospital, farm, or river: each fact lives in a different paper, reported in a different vocabulary. The standard tool for mapping such a fragmented literature is the scoping review — a synthesis method that charts the breadth and structure of evidence rather than pooling statistics, chosen precisely when literatures are too heterogeneous for numerical meta-analysis. But scoping reviews are done by hand: Felipe Schardong et al. screened 3,799 records down to 36 studies of antibiotic-resistance models, and their careful map was static the day it was published.

Automation exists, but for the wrong evidence. TrialPanorama fine-tunes 8-billion-parameter models to search, screen, and summarize 1.66 million clinical-trial records, but it covers human trials in oncology and general medicine, linked to ontologies with no place to record a sequence type or a plasmid backbone. BioGen grounds a language model's interpretation of RNA-seq gene clusters in retrieved literature, cutting hallucination from 0.67 to 0.00 — but it interprets pre-clustered data rather than reading the literature itself, and its rejection-based verification discards exactly the sparse-evidence topics where new resistance mechanisms first surface. Nothing today can read a molecular bacteriology paper and emit an auditable, charted record of which resistance determinant was found in which host, reservoir, and mobile element, at what susceptibility value. This proposal builds and benchmarks that capability, and in doing so attacks the unresolved tension BioGen exposed between verification and coverage.

Core Claim

The claim has three parts. First, automated evidence charting for the antibiotic-resistance literature needs its own evaluation resource: a gold-standard corpus of molecular surveillance papers with span-annotated charting fields, because the entities that matter here — resistance alleles like $bla_{KPC-2}$, sequence types like ST463, plasmid platforms, minimum inhibitory concentration values — appear in no existing review-automation benchmark. Second, a method that forces every charted value to carry a verbatim supporting span from the source paper, and that routes disagreements between two blinded independent agents to a third arbitrator — a direct emulation of the dual-review-plus-arbitration protocol that Juan Li et al. use to control selection bias and that Juan Wang et al. reconcile through third-party arbitration — can hold hallucination near zero without BioGen's coverage loss, provided agents abstain and label a field "no grounded evidence in source" instead of either inventing or silently dropping it. Third, the coverage–precision frontier, not raw accuracy, is the right thing to measure: BioGen's authors report that eliminating hallucinations cost them interpretive coverage, but give no number (no reported result found on Lacuna for BioGen's coverage), and our benchmark makes that tradeoff measurable for the first time.

Research Program

The program has three components: a schema, a corpus, and a method.

The schema is a charting form for molecular resistance surveillance, derived empirically from what the primary literature actually reports. The design is driven by concrete studies: Lingyi Zeng et al. report an IS26 composite transposon carrying $bla_{KPC-2}$ on a conjugative plasmid with an intact Type IV secretion system, so the schema must separate allele, transposon, and plasmid platform; Jie Sheng et al. trace the global phylogeography of E. coli carrying $bla_{NDM-9}$, so geography and sequence type are first-class fields; Philipp Köhler et al. find near-identical $bla_{VIM}$ sequences in patients and local waterways, and Yuting Deng et al. map resistance genes across aquaculture, so reservoir and One Health sector are mandatory fields rather than afterthoughts — the very dimension that Schardong et al. found in only 1 of 36 modeling studies. Quantitative susceptibility values, with units and method, complete the form, following the charting-form discipline of clinical scoping reviews such as Duo Zhang et al..

The corpus starts from Schardong's published funnel: the 3,799-record screening pool and 36 included studies become a screening-recall gold standard and the first charting cohort. We extend it by citation snowballing — the forward-and-backward citation tracking that Ruiyin Li et al. use to catch keyword drift — outward from the empirical anchors in the file, including Dingjie Huang et al. on plasmid-driven Salmonella expansion through food trade and Ying Zhu et al. on ST463 imipenem-nonsusceptible P. aeruginosa, to roughly 150 full-text papers spanning clinical, agricultural, and aquatic reservoirs. Two human annotators chart every paper independently, blinded to each other, with structured arbitration on disagreement; their agreement (Cohen's kappa) sets the human ceiling that any automated system should be judged against, mirroring the dual-review protocol itself.

The method is a span-grounded dual-review pipeline. Two agents — different model families, different retrieval orderings, no shared context — each screen and chart a paper independently. Every charted field must be accompanied by a verbatim character span from the source; a field whose value cannot be tied to a span must be abstained on and logged as evidence-sparse rather than guessed or silently omitted. Disagreements go to a third arbitrator agent that sees only the two charted records and the source, never the agents' reasoning. A field counts as correct only if the value matches the gold chart and the cited span genuinely entails it:

$$\text{GAcc}=\frac{\#\{f:\ \text{value}(f)=\text{gold}(f)\ \wedge\ \text{span}(f)\subseteq\text{source}\ \wedge\ \text{span}(f)\ \text{entails}\ f\}}{\#\text{attempted fields}},\qquad \text{Cov}=\frac{\#\text{attempted fields}}{\#\text{chartable fields}},$$

where $f$ is one slot of the charting schema for one paper, GAcc is grounded accuracy over attempted fields, and Cov is coverage over all fields the human annotators could chart. Reporting GAcc and Cov separately, and sweeping the abstention threshold to trace a coverage–fidelity frontier, is the central measurement idea: it converts BioGen's qualitative "coverage decreased" observation into a curve with an operating point.

Baselines

The naive reference point is a single unconstrained frontier model prompted to chart papers directly, in the spirit of the Gemini 1.5 Flash baseline that BioGen evaluated on Salmonella enterica RNA-seq interpretation, where it showed a hallucination rate of 0.67. We run the same unconstrained setting on our corpus with the same retrieval access; any grounded pipeline should crush it on citation fidelity, so it functions as the floor rather than the bar. The human dual-review charting described above is the ceiling; we do not expect to reach it, only to quantify the distance.

The strongest existing competitor is the TrialPanorama recipe: TrialPanorama fine-tuned Qwen3-8B and Llama3-8B with supervised fine-tuning and reinforcement learning with verifiable rewards, and these specialized 8B models achieved relative improvements of 73.7% in study search and 67.6% in study screening over Llama-3-70B (the file reports only relative gains; no reported result found on Lacuna for TrialPanorama's absolute screening accuracy). We apply their exact recipe to our corpus — same base models, same training data volume per parameter, same tuning budget, same ontology-linking step extended with our resistance schema — so the comparison isolates the domain and the method, not compute. Beating it means higher grounded charting accuracy and citation fidelity at screening recall within two points of its own; if it wins screening but loses charting, our claim narrows honestly to charting and auditability, and we will say so.

The third baseline is the BioGen architecture itself, run unchanged on our corpus: retrieval-augmented generation with a panel of critic agents that reject narratives below a reliability threshold, which achieved hallucination 0.00 and a Semantic Alignment Score of 0.59 on the PRJEB67574 S. enterica dataset. It is the key competitor on the coverage question, because its rejection mechanism is the alternative to our abstention mechanism. Beating it means matching its hallucination rate within 0.05 while achieving strictly higher coverage at every point on the abstention threshold sweep.

The fourth baseline isolates our cheapest ingredient: the "classify, then extract" span model of Danish Pruthi et al., a BERT encoder with a conditional random field that extracts contiguous evidence spans conditioned on document labels, which reached extraction F1 of 46.3 on sentiment evidence (versus about 27.7 for unsupervised attention) and works with as few as 100–200 human-annotated examples. We run it with exactly 200 span annotations from our corpus, matching its reported annotation budget, to test whether classical span extraction plus a rule-based value parser gets most of the way for a fraction of the cost. The final ablation is our own pipeline with the dual-review layer removed — one span-grounded agent, no arbitrator — which isolates whether the blinded-redundancy ingredient, as opposed to span grounding alone, carries the accuracy.

Why not just…?

Why not just commission another manual scoping review? The archetype already exists: Schardong et al. executed a PRISMA-ScR-compliant manual review of resistance models, and it is precisely their findings — 69.4% of models built on deterministic ordinary differential equations, 50% omitting pharmacokinetic/pharmacodynamic dynamics, only one of 36 adopting a One Health perspective — that motivate wanting a continuously refreshable map. Their funnel from 3,799 records to 36 studies took years of blinded human screening, and the result is frozen at 2024. Manual review remains the gold standard for a one-time map; it cannot track emerging threats like the plasmid-borne co-occurrence of mcr-9 with metallo-β-lactamase genes documented by Kewang Hu et al. and Jie Li et al. as they appear. Our proposal does not replace manual review; it benchmarks against it directly and aims to keep its output current between human passes.

Why not just fine-tune a small model on the existing clinical-trial machinery? This is the real threat, and we treat it as the strongest baseline rather than arguing around it. TrialPanorama demonstrated that specialized 8B models beat 70B general models on search and screening, and its recipe may well win our screening task too. Two things it does not provide: its ontologies and record schema have no vocabulary for resistance alleles, sequence types, or mobile genetic elements, so charting accuracy on our schema is untested territory for it; and it produces no span-level audit trail, which is the property that makes a charted claim checkable by a human reviewer in seconds. The head-to-head experiment under identical tuning budget settles this, and if the fine-tuned model matches us on both charting accuracy and citation fidelity, the benchmark contribution stands on its own and we will report the method claim as weakened.

Why not just use BioGen off the shelf? BioGen is the closest existing system, and we build directly on its verification idea and its hallucination metric. But it operates as a post hoc interpreter of pre-clustered transcriptomic modules, not as a reader of surveillance literature, so it cannot screen records or fill a charting schema. More importantly, its authors report that critic-based rejection caused a decrease in interpretive coverage — sparse-literature topics simply go uncharacterized (no reported result found on Lacuna for the size of that coverage loss). Sparse literature is where new resistance biology lives, so a system that cannot say "no grounded evidence found, field left open, here is the abstention log" is structurally blind to emergence. Our coverage–fidelity sweep is designed to show whether abstention recovers what rejection discards.

Why not just prompt a frontier model zero-shot? The evidence against this is direct: the unconstrained Gemini baseline in BioGen hallucinated on two-thirds of its biological statements about AMR gene clusters (rate 0.67), and TrialPanorama found general-purpose 70B models and frontier proprietary models behind specialized small models on review workflows. For evidence synthesis, where a single invented susceptibility value can mislead a stewardship decision, an unaudited generation is worse than no generation.

Proposed Experiments and Decisive Checks

The first experiment builds the corpus and the human ceiling: dual-annotated charting of roughly 150 papers, reporting inter-annotator kappa per schema field, with the screening gold standard inherited from Schardong's 3,799-record pool re-verified by our annotators. The second evaluates screening: every system runs on the full pool, and we report recall against the human-verified inclusion set, with the TrialPanorama-recipe model as the bar. The third evaluates charting: grounded accuracy, citation fidelity (the fraction of charted fields whose spans are verbatim and entailing, the direct descendant of BioGen's hallucination measure), and semantic alignment against the gold chart, for all baselines and both ablations, under matched retrieval access and compute. The decisive fourth check is the coverage–fidelity frontier: sweeping our abstention threshold and BioGen's rejection threshold on the same papers, we ask whether any operating point holds hallucination at or below 0.05 while covering strictly more chartable fields than rejection-based verification, with special attention to the deliberately sparse-evidence stratum — papers reporting rare gene co-occurrences like mcr-9 alongside carbapenemases — where BioGen's architecture should lose the most coverage. The fifth check tests the "living map" claim directly: we withhold the most recent year of the corpus, run the full pipeline on it cold, and measure charting throughput, cost, and agreement with the human chart, demonstrating whether a scoping map can be refreshed in days rather than years. If the span-grounded dual-review pipeline cannot beat the single-agent span-grounded ablation on the disagreement subset, the arbitration ingredient is not earning its cost, and the honest conclusion would be that span grounding alone — the cheapest idea in this proposal — is the contribution worth keeping.

References

  1. Antimicrobial Activity of Colistin Against Contemporary (2015 – 2017) P. aeruginosa and A. baumannii Isolates From a Chinese Surveillance Program
  2. Mapping the landscape of mathematical models for antimicrobial resistance: a scoping review
  3. Developing Large Language Models for Clinical Research Using One Million Clinical Trials
  4. BioGen: An Evidence-Grounded Framework for Interpreting RNA-seq Gene Clusters in Antimicrobial Resistance Research
  5. Diagnosis for Chinese patients with Light Chain Amyloidosis: A scoping review
  6. Evaluation of compliance of CONSORT-CHM formula 2017 in randomized controlled trials of Chinese herbal medicine formulas: protocol of a five-year review
  7. Microbial Characteristics and Genomic Analysis of an ST11 Carbapenem-Resistant Klebsiella pneumoniae Strain Carrying blaKPC−2 Conjugative Drug-Resistant Plasmid
  8. Global geographic and genomic epidemiology analysis of carbapenem-resistant Escherichia coli carrying bla NDM-9
  9. Dissemination of Verona Integron-encoded Metallo-β-lactamase among clinical and environmental Enterobacteriaceae isolates in Ontario, Canada
  10. Global distribution of antimicrobial resistance genes in aquaculture
  11. Application of family-centered care in cardiac surgery: a scoping review
  12. Understanding software architecture erosion: A systematic mapping study
  13. Plasmid-driven clonal expansion of multidrug-resistant monophasic Salmonella Typhimurium in a Global Food Trade Hub
  14. Emergence of ST463 exoU -Positive, Imipenem-Nonsusceptible Pseudomonas aeruginosa Isolates in China
  15. Weakly- and Semi-supervised Evidence Extraction
  16. Molecular characterization of NDM-1-producing carbapenem-resistant E. cloacae complex from a tertiary hospital in Chongqing, China
  17. Molecular antibiotic resistance mechanisms and co-transmission of the mcr-9 and metallo-β-lactamase genes in carbapenem-resistant Enterobacter cloacae complex