Machine learning classification by fitting amplicon sequences to existing OTUs

Courtney R. Armour, Kelly L. Sovacool, William L. Close, Begüm D. Topçuoğlu, Jenna Wiens, Patrick D. Schloss

TL;DR

Microbiome-based diagnostics often fail in the real world because standard "de novo" clustering methods redefine categories every time new data is added, forcing researchers to retrain their models constantly. This paper demonstrates that OptiFit, a tool that fits new sequences into existing clusters, maintains high diagnostic accuracy without the need for retraining, clearing a major hurdle for deploying microbiome-based clinical tools.

What Problem Does This Paper Address?

To understand the gut microbiome, researchers often group similar 16S rRNA gene sequences into Operational Taxonomic Units (OTUs)—essentially "bins" that represent different microbial groups.

The gold standard for creating these bins is de novo clustering, which groups sequences based on their similarity to each other within a specific dataset. While accurate, this method is "unstable." If you get a sample from a new patient, you have to re-cluster all the old data with the new data. This often changes the definitions of the OTUs, meaning the machine learning model you spent months training is suddenly obsolete and must be retrained.

This lack of stability makes it nearly impossible to create a "single, validated model" that can be widely deployed in a clinical setting.

Methods

The researchers compared two primary workflows:

  1. The Standard (OptiClust): All data (training and testing) is clustered together de novo. The model is then trained and tested on these shared OTU definitions.
  2. The Streamlined (OptiFit): Only the training data is clustered de novo. When it’s time to evaluate the model, the new "test" sequences are "fitted" into the existing training OTUs using the OptiFit algorithm.

The team used a Random Forest machine learning model (implemented via the mikropml R package) to classify the samples. They evaluated the quality of the clustering using the Matthews Correlation Coefficient (MCC) and the performance of the model using the Area Under the Receiver Operating Characteristic curve (AUROC).

Datasets

The study utilized a publicly available dataset of 16S rRNA gene sequences from stool samples of 490 subjects:

  • 261 healthy controls (normal or high-risk normal).
  • 229 subjects with Screen Relevant Neoplasia (SRN), which includes both advanced adenomas and carcinomas (colorectal cancer).

The data was randomly split into 80% training and 20% testing sets, and this process was repeated 100 times to ensure the results were robust and not due to a lucky split.

Key Insights and Ideas

  • Stability is the Bridge to the Clinic: The "aha moment" here isn't that OptiFit is better than de novo clustering, but that it is just as good while being significantly more stable. This allows for a "plug-and-play" diagnostic where a model trained once can be used indefinitely.
  • Minimal Data Loss: A common fear with reference-based fitting is that "novel" sequences in new samples will be discarded if they don't fit the reference. However, the authors found that 99.9% of sequences from the test sets successfully mapped to the training OTUs.
  • Reference Quality Matters: Instead of using a generic public database as a reference (which often yields poor quality OTUs), OptiFit uses the study’s own training data as the reference, preserving the high resolution of de novo clustering.

Key Results

  • Cluster Quality: The quality of the OTUs was nearly identical between the two methods. The de novo approach had an average MCC of 0.884, while the OptiFit approach averaged 0.879.
  • Diagnostic Performance: There was no statistically significant difference in the model's ability to detect SRN.
  • During cross-validation, the AUROC was equivalent (p = 0.13).
  • On the held-out test data, the performance remained equivalent (p = 0.38).
  • Efficiency: By using OptiFit, the researchers showed they could classify new samples using a pre-existing model without the computational overhead of reclustering the entire historical dataset.

Limitations and Future Work

While the results are promising for colorectal cancer screening, the authors note a few areas for further exploration:

  • Training Set Size: Future research is needed to determine exactly how many samples are required in a training set to build a "robust" enough reference that can capture the microbial diversity of a broad population.
  • Generalizability: The study focused on SRN; the workflow should be tested across other microbiome-associated diseases (like IBD or metabolic disorders) to ensure the "fit-to-reference" approach holds up.
  • Clinical Implementation: The ultimate goal is a non-invasive, low-cost diagnostic aid. This paper provides the software framework, but the path to a regulated clinical test requires further validation in larger, more diverse cohorts.

Summary updated

Abstract

The ability to use 16S rRNA gene sequence data to train machine learning classification models offers the opportunity to diagnose patients based on the composition of their microbiome. In some applications, the taxonomic resolution that provides the best models may require the use of <i>de novo</i> operational taxonomic units (OTUs) whose composition changes when new data are added. We previously developed a new reference-based approach, OptiFit, that fits new sequence data to existing <i>de novo</i> OTUs without changing the composition of the original OTUs. While OptiFit produces OTUs that are as high quality as <i>de novo</i> OTUs, it is unclear whether this method for fitting new sequence data into existing OTUs will impact the performance of classification models relative to models trained and tested only using <i>de novo</i> OTUs. We used OptiFit to cluster sequences into existing OTUs and evaluated model performance in classifying a dataset containing samples from patients with and without colonic screen relevant neoplasia (SRN). We compared the performance of this model to standard methods including <i>de novo</i> and database-reference-based clustering. We found that using OptiFit performed as well or better in classifying SRNs. OptiFit can streamline the process of classifying new samples by avoiding the need to retrain models using reclustered sequences. IMPORTANCE There is great potential for using microbiome data to aid in diagnosis. A challenge with <i>de novo</i> operational taxonomic unit (OTU)-based classification models is that 16S rRNA gene sequences are often assigned to OTUs based on similarity to other sequences in the dataset. If data are generated from new patients, the old and new sequences must be reclustered to OTUs and the classification model retrained. Yet there is a desire to have a single, validated model that can be widely deployed. To overcome this obstacle, we applied the OptiFit clustering algorithm to fit new sequence data to existing OTUs allowing for reuse of the model. A random forest model implemented using OptiFit performed as well as the traditional reassign and retrain approach. This result shows that it is possible to train and apply machine learning models based on OTU relative abundance data that do not require retraining or the use of a reference database.
BibTeX
@inproceedings{armour2023machine,
  author = {Courtney R. Armour and Kelly L. Sovacool and William L. Close and Begüm D. Topçuoğlu and Jenna Wiens and Patrick D. Schloss},
  title = {{Machine learning classification by fitting amplicon sequences to existing OTUs}},
  booktitle = {mSphere},
  year = {2023},
  doi = {10.1128/msphere.00336-23},
}
Figure