Tracking SARS-CoV-2 variants of concern in wastewater: an assessment of nine computational tools using simulated genomic data

Steven G. Sutcliffe, Susanne A. Kraemer, Isaac Ellmen, Jennifer J. Knapp, Alyssa K. Overton, Delaney Nash, Jozef I. Nissimov, Trevor C. Charles, David Dreifuss, Ivan Topolsky, Pelin Icer Baykal, Lara Fuhrmann, Kim Philipp Jablonski, Niko Beerenwinkel, Joshua I. Levy, Abayomi S. Olabode, Devan Becker, Gopi Gugan, Erin Britnell, Art F. Y. Poon, Renan Valieris, Rodrigo D. Drummond, Alexandre Defelicibus, Emmanuel Dias‐Neto, Rafael Rosales, Israel Tojal da Silva, Aspasia Orfanou, Fotis Psomopoulos, Nikolaos Pechlivanis, Lenore Pipes, Zihao Chen, Jasmijn A. Baaijens, Michael Baym, B. Jesse Shapiro

Abstract

Wastewater-based surveillance (WBS) has emerged as a critical public health tool for monitoring pathogen genetic diversity at scale. However, deconvolving complex mixtures of viral lineages from short-read sequencing data remains a significant computational challenge. This study benchmarks nine tools—Alcov, Basic, Freyja, Gromstole, LCS, Lineagespot, Pipes, VLQ, and V-pipe—using 100 simulated wastewater samples containing SARS-CoV-2 BA.1, BA.2, and Delta lineages, along with recombinants and novel synthetic lineages. While most tools proved robust to variations in sequencing depth and read length, reliability was highest for variants above a 5% abundance threshold.

Introduction

Wastewater-based surveillance (WBS) provides a non-invasive, community-level view of viral spread. While qPCR measures total viral load, genomic sequencing is required to identify specific Variants of Concern (VOCs). Wastewater samples are inherently "genomic soups," containing fragments from numerous individuals and lineages. Deconvolving these mixtures to estimate relative abundances is a complex task that requires specialized computational tools. Until now, there hasn't been a comprehensive, independent comparison of how these different tools handle "messy" real-world conditions like low coverage or the appearance of entirely new, unclassified lineages.

Methodology

The researchers conducted a semi-blind benchmark involving nine computational tools, each employing different algorithmic strategies such as linear modeling, phylogenetic placement, and expectation-maximization.

Table 1: Summary of the nine computational tools assessed in the benchmark, including their underlying methods and database dependencies.

The benchmark used 100 simulated samples designed to test tool performance against varying sequencing depths, read lengths, and the presence of "edge cases" like recombinants and unclassified synthetic lineages. These tools represent a variety of approaches, from linear regression (Alcov, Gromstole) to phylogenetic placement (Freyja) and pseudoalignment (VLQ).

Results and Performance

Most tools demonstrated high accuracy in identifying lineages and estimating their relative abundances, achieving high F1 scores and low Root-Mean-Square Error (RMSE). Performance was particularly strong for variants present at frequencies above 5%. Below this threshold, the signal-to-noise ratio often led to increased false positives or quantification errors.

Figure 1: Comparison of predicted lineage relative abundances across the nine tools for BA.1, BA.2, and Delta lineages.

Top-performing tools, including Freyja, Alcov, and V-pipe, showed consistent results across various simulated conditions. Notably, the tools were generally resilient to "bad" data, such as low coverage or shorter read lengths, suggesting that cost-saving measures in sequencing may not severely compromise surveillance quality. However, some tools like Lineagespot (designed for presence/absence) and Basic (optimized for pre-Omicron variants) showed higher error rates in this specific benchmark.

Handling Complexity and Novelty

The study also explored how tools handle "unknown" data. When presented with a novel synthetic lineage, most tools correctly categorized it as "Other" or "Unassigned." However, the presence of such unknown lineages did slightly increase the error in quantifying known variants. Recombinants (e.g., Delta-Omicron) proved more challenging, with some tools identifying the recombinant specifically and others splitting the abundance between the parent lineages.

Conclusion and Future Directions

This assessment highlights the maturity and robustness of current WBS deconvolution tools. While tool choice should be tailored to specific goals—such as presence/absence vs. precise quantification—most methods are capable of providing reliable public health data even under suboptimal sequencing conditions.

Future work should focus on fully automated, independent benchmarking and the development of tools that can better distinguish between closely related sub-lineages as the virus continues to evolve convergently. The authors also note that real-world wastewater is even more complex than these simulations, containing higher levels of degradation and potential contamination from other coronaviruses.

Summary updated

Abstract

Abstract Wastewater-based surveillance (WBS) is an important epidemiological and public health tool for tracking pathogens across the scale of a building, neighbourhood, city, or region. WBS gained widespread adoption globally during the SARS-CoV-2 pandemic for estimating community infection levels by qPCR. Sequencing pathogen genes or genomes from wastewater adds information about pathogen genetic diversity which can be used to identify viral lineages (including variants of concern) that are circulating in a local population. Capturing the genetic diversity by WBS sequencing is not trivial, as wastewater samples often contain a diverse mixture of viral lineages with real mutations and sequencing errors, which must be deconvoluted computationally from short sequencing reads. In this study we assess nine different computational tools that have recently been developed to address this challenge. We simulated 100 wastewater sequence samples consisting of SARS-CoV-2 BA.1, BA.2, and Delta lineages, in various mixtures, as well as a Delta-Omicron recombinant and a synthetic “novel” lineage. Most tools performed well in identifying the true lineages present and estimating their relative abundances, and were generally robust to variation in sequencing depth and read length. While many tools identified lineages present down to 1% frequency, results were more reliable above a 5% threshold. The presence of an unknown synthetic lineage, which represents an unclassified SARS-CoV-2 lineage, increases the error in relative abundance estimates of other lineages, but the magnitude of this effect was small for most tools. The tools also varied in how they labelled novel synthetic lineages and recombinants. While our simulated dataset represents just one of many possible use cases for these methods, we hope it helps users understand potential sources of noise or bias in wastewater sequencing data and to appreciate the commonalities and differences across methods.
BibTeX
@inproceedings{sutcliffe2024tracking,
  author = {Steven G. Sutcliffe and Susanne A. Kraemer and Isaac Ellmen and Jennifer J. Knapp and Alyssa K. Overton and Delaney Nash and Jozef I. Nissimov and Trevor C. Charles and David Dreifuss and Ivan Topolsky and Pelin Icer Baykal and Lara Fuhrmann and Kim Philipp Jablonski and Niko Beerenwinkel and Joshua I. Levy and Abayomi S. Olabode and Devan Becker and Gopi Gugan and Erin Britnell and Art F. Y. Poon and Renan Valieris and Rodrigo D. Drummond and Alexandre Defelicibus and Emmanuel Dias‐Neto and Rafael Rosales and Israel Tojal da Silva and Aspasia Orfanou and Fotis Psomopoulos and Nikolaos Pechlivanis and Lenore Pipes and Zihao Chen and Jasmijn A. Baaijens and Michael Baym and B. Jesse Shapiro},
  title = {{Tracking SARS-CoV-2 variants of concern in wastewater: an assessment of nine computational tools using simulated genomic data}},
  booktitle = {Microbial},
  year = {2024},
  doi = {10.1099/mgen.0.001249},
}
Figure