Bacterial species identification using MALDI-TOF mass spectrometry and machine learning techniques: A large-scale benchmarking study

Thomas Mortier, Anneleen D. Wieme, Peter Vandamme, Willem WaegemanComputational 2021 · DOI

TL;DR

This study benchmarks a wide range of machine learning techniques for bacterial species identification using MALDI-TOF MS on an unprecedented scale, involving nearly 100,000 spectra and over 1000 species. It reveals that while current methods achieve acceptable identification rates across novel replicates, strains, and species, performance drops compared to smaller-scale studies, and taxonomic information isn't always well-preserved. The paper also introduces the use of Monte Carlo dropout neural networks for effective novel species detection.

What Problem Does This Paper Address?

Accurately identifying bacterial species is crucial for fields ranging from clinical diagnostics to environmental monitoring. MALDI-TOF mass spectrometry (MS) has become a popular, cost-effective method for this, generating unique spectral "fingerprints" for different bacteria. Machine learning (ML) algorithms are increasingly used to interpret these spectra. However, most existing ML studies in this area are limited by small datasets and a narrow range of species, often failing to distinguish between different identification challenges: identifying new samples of known strains (novel biological replicates), identifying new strains of known species (novel strains), or identifying entirely new species (novel species). This lack of large-scale, comparative benchmarking hinders our understanding of how well current ML methods generalize and where improvements are needed, especially concerning the "black-box" nature of commercial identification software.

Methods

The researchers benchmarked various ML algorithms, including traditional ones like Random Forests (RF), Logistic Regression (LR), Support Vector Classifiers (LSVC), and K-Nearest Neighbors (KNN), alongside a 1D Convolutional Neural Network (1DCNN). They explored two main classification approaches:

  • Flat Classification: Standard multi-class classification where each species is treated independently, ignoring hierarchical taxonomic relationships.
  • Hierarchical Classification: Methods that leverage phylogenetic information (family, genus, species) to guide classification, using a "local classifier per parent node" approach.

For the challenging "novel species detection" scenario, they adapted neural networks with Monte Carlo Dropout. This technique, typically used in computer vision, approximates Bayesian inference by applying dropout during both training and testing. By analyzing the uncertainty in predictions, it helps identify samples that are significantly different from the training data, indicating a potential novel species.

Preprocessing steps included binning (dividing spectra into fixed-size m/z intervals), baseline correction (removing background noise), and total ion current normalization (rescaling spectra).

Datasets

The study utilized three primary datasets:

  • Global Dataset (GD): A large, historical collection of spectra from over 2400 reference strains representing more than 1000 bacterial species, curated by Ghent University. This dataset was split into training (GDtrain), validation (GDval), and testing (GDtest) sets, ensuring no strain overlap between sets for the "novel strains" scenario.
  • Lyopreservation Dataset (LYO1 & LYO2): A smaller dataset used to evaluate the "novel biological replicates" scenario. LYO1 was used for training, and LYO2 for testing.
  • Additional Test Dataset (TEST): A dataset derived from a recent microbial diversity project, used specifically for the "novel species detection" scenario. This dataset was designed to contain species not present in the GD.

Key Insights and Ideas

  • Scale Matters: Benchmarking on a large dataset (100,000+ spectra, 1000+ species) reveals performance differences not apparent in smaller studies.
  • Hierarchical Information is Underutilized: Taxonomic hierarchy doesn't seem to be inherently well-preserved in MALDI-TOF MS data, as hierarchical classifiers didn't consistently outperform flat classifiers at the species level.
  • Novel Species Detection is a Distinct Challenge: Identifying entirely new species requires specialized techniques beyond standard classification, highlighting the importance of uncertainty estimation.
  • Monte Carlo Dropout Shows Promise: This technique offers a robust way to quantify prediction uncertainty, proving effective for novel species detection.
  • Preprocessing Trade-offs: While binning and normalization are crucial, their impact varies across different ML models (e.g., LR struggled with ALS+TIC).

Key Results

  • Novel Strains & Replicates:
    • KNN achieved the highest accuracy (84.78%) for novel strains using flat classification.
  • LSVC achieved the highest accuracy (96.15%) for novel biological replicates.
  • 1DCNNs did not outperform traditional methods like KNN and LSVC in these scenarios, despite their potential for complex feature learning.
  • Hierarchical classifiers showed higher accuracy at the family level compared to species level, suggesting a better preservation of broader taxonomic information.
  • Novel Species Detection:
    • 1DCNN with a high dropout rate (0.8) achieved near-perfect performance (AUROC of 0.9997, AUPR of 0.9975) in detecting novel species.
  • Principal Component Analysis (PCA) on the penultimate layer of the 1DCNN clearly separated in-distribution from out-of-distribution samples, especially when visualized with total uncertainty.
  • KNN performed well but was outperformed by the 1DCNN with Monte Carlo dropout.

Limitations and Future Work

  • Taxonomic Information: The study suggests that taxonomic information is not strongly encoded in MALDI-TOF MS spectra, limiting the benefit of hierarchical classification at the species level. Further research could explore how to better extract or leverage this information.
  • Novel Species Detection Methods: While Monte Carlo dropout showed promise, the authors suggest a more extensive study comparing various uncertainty estimation methods, particularly those that can differentiate between aleatoric (inherent noise) and epistemic (model uncertainty) uncertainty.
  • Computational Cost: The sheer scale of the dataset made some advanced methods (like the PIKE kernel mentioned) computationally infeasible. Future work could focus on scalable methods.
  • Generalizability: While the study uses a large dataset, further validation across different labs and instruments could be beneficial to assess the impact of batch effects.

Summary updated

Abstract

Today machine learning methods are commonly deployed for bacterial species identification using MALDI-TOF mass spectrometry data. However, most of the studies reported in literature only consider very traditional machine learning methods on small datasets that contain a limited number of species. In this paper we present benchmarking results on an unprecedented scale for a wide range of machine learning methods, using datasets that contain almost 100,000 spectra and more than 1000 different species. The size and the diversity of the data allow to compare three important identification scenarios that are often not distinguished in literature, i.e., identification for novel biological replicates, novel strains and novel species that are not present in the training data. The results demonstrate that in all three scenarios acceptable identification rates are obtained, but the numbers are typically lower than those reported in studies with a more limited analysis. Using hierarchical classification methods, we also demonstrate that taxonomic information is in general not well preserved in MALDI-TOF mass spectrometry data. For the novel species scenario, we apply for the first time neural networks with Monte Carlo dropout, which have shown to be successful in other domains, such as computer vision, for the detection of novel species.
BibTeX
@inproceedings{mortier2021bacterial,
  author = {Thomas Mortier and Anneleen D. Wieme and Peter Vandamme and Willem Waegeman},
  title = {{Bacterial species identification using MALDI-TOF mass spectrometry and machine learning techniques: A large-scale benchmarking study}},
  booktitle = {Computational},
  year = {2021},
  doi = {10.1016/j.csbj.2021.11.004},
}
Figure