FDA-ARGOS: A Public Quality-Controlled Genome Database Resource for Infectious Disease Sequencing Diagnostics and Regulatory Science Research

Heike Sichtig, Timothy D. Minogue, Yi Yan, Christopher P. Stefan, Adrienne T. Hall, Luke J. Tallon, Lisa Sadzewicz, Suvarna Nadendla, William Klimke, Eneida Hatcher, Martin Shumway, Dayanara Lebron Aldea, Jonathan Allen, Jeffrey W. Koehler, Tom Slezak, Stephen Lovell, Randal J. Schoepp, Uwe ScherfbioRxiv 2018 · DOI

TL;DR

The FDA has developed FDA-ARGOS, a public database of high-quality, quality-controlled microbial genomes. This resource aims to fill critical gaps in existing public databases, enabling more accurate infectious disease diagnostics and streamlining the validation of next-generation sequencing (NGS) technologies.

What Problem Does This Paper Address?

Next-generation sequencing (NGS) is revolutionizing infectious disease diagnostics, offering the potential for rapid and comprehensive identification of pathogens. However, the accuracy and reliability of these diagnostics heavily depend on the quality and completeness of reference genome databases. Existing public databases often suffer from incomplete or low-quality genomes, leading to misidentifications and hindering the development and regulatory approval of new diagnostic tools. This paper addresses the critical need for a curated, high-quality reference genome resource to support the advancement of infectious disease NGS diagnostics and regulatory science.

Methods

The authors describe the development and implementation of the FDA-ARGOS database, focusing on its quality control (QC) metrics and data requirements. Key aspects include:

  • Hybrid Sequencing Approach: Utilizing both long-read (PacBio) and short-read (Illumina) sequencing technologies to generate high-quality bacterial and viral genomes.
  • De Novo Assembly: Employing a suite of assembly tools (SPAdes, Canu, HGAP, Celera Assembler) followed by manual curation and polishing (Pilon) to produce robust genome assemblies.
  • Rigorous Quality Control: Establishing stringent criteria for "regulatory-grade" genomes, including minimum coverage (95% with 20X depth), NCBI taxonomy-specific average nucleotide identity (ANI) thresholds, and adherence to a minimum of 10 critical metadata fields.
  • Composite Reference Method (C-RM): Proposing a novel validation strategy that combines in silico sequence comparison using FDA-ARGOS with a wet-lab validation challenge. This approach aims to reduce the burden of empirical validation for ID-NGS diagnostic assays.

Datasets

The paper utilizes several datasets for its analyses and database construction:

  • FDA-ARGOS Database: A newly established database containing quality-controlled microbial genomes. The initial subset presented includes 487 genomes (88.3% bacteria, 11.1% viruses, 0.6% eukaryotes).
  • Publicly Available Genomes: Data from NCBI GenBank (2013 and 2018) were used for comparative quality assessments.
  • Clinical Samples:
    • Fifteen de-identified human serum samples positive for Ebola virus (EBOV) Makona from Sierra Leone.
  • Ten de-identified human serum samples suspected of being Bundibugyo virus positive from the Democratic Republic of Congo.
  • Mock Clinical Samples:
    • A mock clinical Enterococcus avium sample (isolate SAMN04327393) used for metagenomic and isolate shotgun sequencing.
  • Mock clinical samples for EBOV and Bundibugyo virus used for MIPS assay evaluation.

Key Insights and Ideas

  • The "Garbage In, Garbage Out" Principle in Genomics: The paper strongly emphasizes that the quality of reference genomes directly impacts the accuracy of NGS-based diagnostics. Using incomplete or low-quality genomes can lead to misidentifications or a complete failure to identify pathogens.
  • FDA-ARGOS as a Crucial Gap Filler: The database is shown to significantly improve the ability to classify microbial reads, particularly for organisms where public databases are sparse or contain low-quality data.
  • The Power of In Silico Validation: The proposed Composite Reference Method (C-RM), which integrates in silico comparison with FDA-ARGOS, offers a more efficient and cost-effective approach to validating ID-NGS assays compared to purely empirical methods.
  • Defining "Regulatory-Grade" Genomes: The paper clearly outlines the criteria for what constitutes a high-quality, regulatory-grade genome, providing a benchmark for future genomic data generation and curation.

Key Results

  • Improved Read Classification: In the Enterococcus avium use case, adding FDA-ARGOS genomes to reference databases increased the number of classified E. avium reads by an average of 3829 to 840 (MegaBLAST) and 29 to 0 (Kraken) when FDA-ARGOS genomes were absent. For isolate shotgun sequencing, FDA-ARGOS contributed 84-96% of the total E. avium reads classified.
  • Reduced Misclassification: Using curated FDA-ARGOS genomes helped to avoid misclassification of reads, as seen with E. avium where top hits from NCBI Nt (containing lower-quality genomes) included potential contaminants like Bos Taurus.
  • Validation of In Silico Approach: For Ebola and Bundibugyo viruses, in silico comparison using FDA-ARGOS showed high concordance with benchmark real-time PCR assays, demonstrating its utility for validation.
  • MIPS Assay Performance: The MIPS assay showed promising diagnostic performance, with positive predictive values of 97.4% and negative predictive values of 90% for EBOV at or above the limit of detection.

Limitations and Future Work

  • Completeness of the Database: While FDA-ARGOS is growing, it is still a developing resource. The authors acknowledge that continued population and expansion are required to cover the full spectrum of infectious microorganisms.
  • Need for Empirical Validation: The paper highlights that while in silico validation is powerful, complete in silico validation alone is unrealistic for clinical trials without some comparison to a gold standard reference method (e.g., real-time PCR).
  • Future Work:
    • Continued population and curation of the FDA-ARGOS database.
  • Development of pathways for external genome qualification to streamline and expand the database.
  • Applying regulatory-grade standards to unculturable and emerging pathogens.
  • Utilizing deep learning methods and artificial intelligence for genome qualification.
  • Soliciting collaborators to contribute unique and hard-to-source microbial isolates for gap-filling.

Summary updated

Abstract

ABSTRACT Infectious disease next generation sequencing (ID-NGS) diagnostics are on the cusp of revolutionizing the clinical market. To facilitate this transition, FDA proactively invested in tools to support innovation of emerging technologies. FDA and collaborators established a publicly available database, FDA dAtabase for Regulatory-Grade micrObial Sequences (FDA-ARGOS), as a tool to fill reference database gaps with quality-controlled genomes. This manuscript discusses quality control metrics for the proposed FDA-ARGOS genomic resource and outlines the need for quality-controlled genome gap filling in the public domain. Here, we also present three case studies showcasing potential applications for FDA-ARGOS in infectious disease diagnostics, specifically: assay design, reference database and in silico sequence comparison in combination with representative microbial organism wet lab testing; a novel composite validation strategy for ID-NGS diagnostics. The use of FDA-ARGOS as an in silico comparator tool could reduce the burden for completing ID-NGS clinical trials. In addition, use cases identifying Enterococcus avium and Ebola virus (Zaire ebolavirus variant Makona) demonstrate the utility of FDA-ARGOS as a reference database for independent performance validation of new tests and for documenting how one would use this database as an in silico sequence target comparator tool for ID-NGS validation, respectively.
BibTeX
@misc{sichtig2018fda,
  author = {Heike Sichtig and Timothy D. Minogue and Yi Yan and Christopher P. Stefan and Adrienne T. Hall and Luke J. Tallon and Lisa Sadzewicz and Suvarna Nadendla and William Klimke and Eneida Hatcher and Martin Shumway and Dayanara Lebron Aldea and Jonathan Allen and Jeffrey W. Koehler and Tom Slezak and Stephen Lovell and Randal J. Schoepp and Uwe Scherf},
  title = {{FDA-ARGOS: A Public Quality-Controlled Genome Database Resource for Infectious Disease Sequencing Diagnostics and Regulatory Science Research}},
  year = {2018},
  doi = {10.1101/482059},
  howpublished = {bioRxiv},
}
Figure