Abstract
The volume of biological data being generated by the scientific community is growing exponentially, reflecting technological advances and research activities. The National Institutes of Health's (NIH) Sequence Read Archive (SRA), which is maintained by the National Center for Biotechnology Information (NCBI) at the National Library of Medicine (NLM), is a rapidly growing public database that researchers use to drive scientific discovery across all domains of life. This increase in available data has great promise for pushing scientific discovery but also introduces new challenges that scientific communities need to address. As genomic datasets have grown in scale and diversity, a parade of new methods and associated software have been developed to address the challenges posed by this growth. These methodological advances are vital for maximally leveraging the power of next-generation sequencing (NGS) technologies. With the goal of laying a foundation for evaluation of methods for petabyte-scale sequence search, the Department of Energy (DOE) Office of Biological and Environmental Research (BER), the NIH Office of Data Science Strategy (ODSS), and NCBI held a virtual codeathon 'Petabyte Scale Sequence Search: Metagenomics Benchmarking Codeathon' on September 27 - Oct 1 2021, to evaluate emerging solutions in petabyte scale sequence search. The codeathon attracted experts from national laboratories, research institutions, and universities across the world to (a) develop benchmarking approaches to address challenges in conducting large-scale analyses of metagenomic data (which comprises approximately 20% of SRA), (b) identify potential applications that benefit from SRA-wide searches and the tools required to execute the search, and (c) produce community resources i.e. a public facing repository with information to rebuild and reproduce the problems addressed by each team challenge.
The Scaling Challenge
The NIH’s Sequence Read Archive (SRA) has exploded to over 38 petabytes of data, making traditional sequence search tools like BLAST impractically slow for full-database queries. This massive, rapidly growing public database holds the key to countless discoveries, such as tracking pathogen spread, discovering novel viruses, and understanding complex microbial ecosystems. However, finding specific genetic sequences across this 38+ petabyte haystack is incredibly difficult.
Currently, researchers often rely on text-based metadata searches, but metadata is frequently incomplete or inconsistent. The alternative—sequence-based search—is computationally expensive. While faster, index-based tools (using k-mers) exist, they often sacrifice sensitivity. The community lacks standardized ways to benchmark these trade-offs and lacks scalable, cloud-native pipelines to execute these searches efficiently.
Methods
The codeathon divided experts into four teams to address specific challenges:
- Team 1 (High-Throughput Screening): Developed the
bothieSnakemake pipeline to evaluate fast, k-mer-based sketching tools (e.g., sourmash, Mash, Metagraph) for finding long query sequences within metagenomes. - Team 2 (Contig Containment): Created a benchmarking pipeline to evaluate tools like BLAST, MMseqs2, and Minimap2 in identifying "containment" relationships between assembled sequences.
- Team 3 (Hard-to-Annotate Sequences): Developed
BAGEL, a Nextflow-based test harness for standardizing the benchmarking of sequence annotation tools in difficult scenarios like translated searches. - Team 4 (BLASTing the SRA): Built a hybrid cloud pipeline using Pebblescout for fast pre-filtering and ElasticBLAST for sensitive alignment of the resulting hits.
Datasets and Metagenomics Focus
The primary target was the SRA, with a specific focus on metagenomic data, which represents a substantial portion of the archive.
Other datasets included curated metagenomes from the National Microbiome Data Collaborative (NMDC) and time-series human gut metagenomes. Specific queries included the Microcystin toxin gene cluster and SARS-CoV-2.
Key Insights and Results
- Hybrid Search Necessity: Sensitive alignment across 38 petabytes is impossible. The future relies on a two-step process: ultra-fast index-based filtering followed by high-sensitivity alignment.
- I/O Bottlenecks: In cloud environments, moving data is harder than analyzing it. Team 4 found that while Pebblescout filtered data in seconds and ElasticBLAST aligned 60 billion bases in an hour, downloading the filtered datasets took ~20 hours.
- Benchmarking Standardization: There is a critical need for community-supported test harnesses to objectively compare tool performance.
- Tool Agreement: Team 2 identified 155,372 contig containments in a marine dataset and found high agreement between BLAST and MMseqs2, while synteny-based methods were more conservative.
Limitations and Future Directions
Transitioning to cloud-based bioinformatics involves steep learning curves for workflow managers. The primary limitation remains the network bottleneck for data downloads. Future work will focus on bypassing this via highly parallel cloud functions to process data directly in the cloud and expanding benchmarks to include more complex genetic elements like non-coding RNA and transposable elements.