TL;DR
This study benchmarks nine computational tools for identifying SARS-CoV-2 lineages in wastewater using simulated genomic data. The findings show that most tools perform well, even with low sequencing depth and short reads, but lineage detection becomes more reliable above a 5% abundance threshold. The presence of unknown or recombinant lineages can introduce minor errors in abundance estimates, highlighting the need for careful interpretation of results.
What Problem Does This Paper Address?
Wastewater-based surveillance (WBS) has become a crucial public health tool for tracking pathogens like SARS-CoV-2. While qPCR can estimate overall viral load, sequencing wastewater offers a deeper look into the genetic diversity of circulating viruses, including identifying specific lineages and variants of concern (VOCs). However, wastewater samples are complex mixtures of viral genetic material, often degraded and containing sequencing errors. Deconvoluting this mixture to accurately identify and quantify different viral lineages computationally is a significant challenge. Numerous tools have been developed to address this, but their comparative performance, especially under realistic wastewater conditions, was unclear. This paper aims to fill that gap by evaluating nine leading computational tools using a simulated dataset that mimics the complexities of real-world wastewater samples.
Methods
The study employed a simulated dataset to benchmark nine different computational tools designed for SARS-CoV-2 lineage deconvolution from wastewater. The simulation involved creating 100 wastewater samples with varying mixtures of SARS-CoV-2 lineages (BA.1, BA.2, Delta), including a Delta-Omicron recombinant and a synthetic "novel" lineage. These samples were generated with simulated sequencing errors, variable coverage depths, and different read lengths to mimic real-world wastewater sequencing data. The nine tools were then applied to this dataset, with each team running their tool according to their own best practices. The performance of each tool was evaluated based on its ability to correctly identify lineages (F1-score) and estimate their relative abundances (Root-Mean-Square Error - RMSE).
Datasets
The primary dataset used in this study was a newly generated simulated dataset of 100 wastewater samples. These samples were constructed to represent various mixtures of SARS-CoV-2 lineages (BA.1, BA.2, Delta), a Delta-Omicron recombinant, and a synthetic "novel" lineage. The simulation incorporated realistic factors such as random sequencing errors, variable sequencing depth (low and high coverage), and different read lengths (150 bp and 250 bp). This simulated dataset serves as a benchmark for evaluating the performance of the computational tools, differing from real-world datasets by offering a known ground truth for comparison.
Key Insights and Ideas
- Robustness to Sequencing Imperfections: Most tools demonstrated a surprising resilience to common sequencing challenges like low depth and short read lengths, suggesting that cost-saving measures in sequencing might be feasible without drastically compromising lineage identification.
- The 5% Threshold Matters: While some tools could detect lineages present at as low as 1% abundance, the reliability of these identifications (measured by F1-score) significantly improved when the detection threshold was raised to 5%. This implies that low-frequency lineages (<5%) should be interpreted with caution.
- Unknowns Introduce Noise: The presence of an unclassified "synthetic" lineage, even if not a known variant, demonstrably increased the error in relative abundance estimates for other lineages. This highlights the challenge of accurately quantifying known lineages when novel or uncharacterized genetic material is present.
- Recombinants and Novelties are Tricky: Tools varied considerably in how they handled simulated recombinant and synthetic lineages. Some correctly identified them as "other" or unassigned, while others attempted to classify them as known lineages or split their calls between parental lineages. This points to an ongoing challenge in accurately classifying novel viral genetic structures.
Key Results
- General Performance: Most of the nine evaluated tools performed well in identifying the true lineages present and estimating their relative abundances.
- Lineage Detection Threshold: Tools showed improved performance (higher F1-scores) when the detection threshold for a lineage was set at 5% abundance compared to 1%.
- Impact of Synthetic Lineage: The presence of a synthetic lineage increased the RMSE for relative abundance estimates across most tools, though the magnitude of this effect was generally small.
- Specific Tool Performance:
- Lineagespot and Basic showed relatively lower F1-scores and higher RMSEs, with Lineagespot's design focused on presence/absence rather than precise abundance.
- The Basic approach struggled to distinguish between BA.1 and BA.2 in single-lineage samples.
- Handling of Novelties:
- Most tools identified the synthetic lineage as "Other."
- Freyja placed the synthetic lineage near the root as a deep-branching 'B' lineage.
- Pipes and VLQ showed instances of "false-positive" identification of Delta or BA.1 for the synthetic lineage.
- For the Delta-Omicron recombinant, several tools split calls between Delta and Omicron, while others identified it as "XS" or unassigned.
Limitations and Future Work
The authors acknowledge several limitations:
- Simulated vs. Real Data: The simulated dataset, while comprehensive, represents a model use case and may not fully capture the extreme complexity of real-world wastewater samples.
- Semi-Blind Trial: The benchmark was a semi-blind trial where each team ran their own tool. A more stringent comparison would involve a single group running all tools to assess user-friendliness and reproducibility more directly.
- Limited Scope: The study focused on SARS-CoV-2 whole-genome sequencing with short reads and did not evaluate lineage-agnostic approaches or long-read sequencing extensively.
- Evolving Landscape: The study reflects the state of tools and questions from 2021-2022. New variants and computational methods have emerged since then.
Future work could involve:
- Expanding benchmarking to include more diverse and complex simulated datasets.
- Developing standardized protocols for running and evaluating wastewater sequencing tools.
- Investigating the performance of tools with long-read sequencing data.
- Exploring lineage-agnostic approaches for identifying novel or unclassified viral sequences.
- Applying these benchmarks to real-world wastewater samples to validate findings.
Summary updated