MassSpecGym: A benchmark for the discovery and identification of molecules

Roman Bushuiev, Anton Bushuiev, Niek F. de Jonge, Adamo Young, Fleming Kretschmer, Raman Samusevich, Janne Heirman, Fei Wang, Luke Zhang, Kai Dührkop, Marcus Ludwig, Nils A. Haupt, Apurva Kalia, Corinna Brungs, Robin Schmid, Russell Greiner, Bo Wang, David S. Wishart, Li-Ping Liu, Juho Rousu, Wout Bittremieux, Hannes Rost, Tytus D. Mak, Soha Hassoun, Florian Huber, Justin J. J. van der Hooft, Michael A. Stravs, Sebastian Böcker, Josef Sivic, Tomáš Pluskal

TL;DR

MassSpecGym is a new, large-scale benchmark designed to bring the "AlphaFold moment" to small molecule identification. By providing a curated dataset of 231,000 high-quality mass spectra and a rigorous evaluation framework that prevents chemical "data leakage," it standardizes the challenge of decoding molecular structures from their fragment fingerprints.

What Problem Does This Paper Address?

Identifying the millions of small molecules in biological and environmental samples is a cornerstone of modern medicine and ecology. The primary tool for this is tandem mass spectrometry (MS/MS), which breaks a molecule into fragments and records their mass-to-charge ratios as a spectrum.

However, interpreting these spectra is a "grand challenge." Even for experts, decoding a molecular graph from a spectrum is incredibly difficult, leaving over 90% of acquired data uninterpreted. While machine learning (ML) is the obvious solution, the field has been held back by:

  1. Data Heterogeneity: Spectra look different depending on the instrument (e.g., Orbitrap vs. QTOF) and the energy used to break the molecule.
  2. Data Leakage: Standard ways of splitting data often put "near-duplicate" molecules in both the training and test sets, leading to over-optimistic performance that fails in the real world.
  3. Lack of Standardization: Researchers use different datasets, metrics, and preprocessing steps, making it impossible to compare models fairly.

Methods

MassSpecGym transforms the process of scientific discovery into three well-defined ML challenges:

  1. De novo molecule generation: Predicting a 2D molecular graph directly from a spectrum.
  2. Molecule retrieval: Ranking candidate molecules from a database to find the best match for a query spectrum.
  3. Spectrum simulation: The inverse problem—predicting what a spectrum will look like for a given molecular graph.
Figure 1: MassSpecGym provides three challenges for benchmarking the discovery and identifi- cation of new molecules from MS/MS spectra. The provided challenges abstract the process of scientific discovery from biological and environmental samples into well-defined machine learning problems.

The authors introduced a novel MCES-based data split. Instead of just ensuring the exact same molecule isn't in both sets, they use the Maximum Common Edge Subgraph (MCES) distance to ensure that test molecules are chemically distinct (at least 10 "edits" away) from anything the model saw during training. This forces models to learn general chemical principles rather than just memorizing structural motifs.

Datasets

The benchmark is built on the largest publicly available collection of high-quality labeled MS/MS spectra.

  • MassSpecGym (Core): 231,104 spectra representing 29,000 unique molecules. This includes a significant portion of newly measured in-house data.
  • Comparison: As shown in the table below, MassSpecGym offers a massive step up in scale and quality compared to previous benchmarks like MIST CANOPUS.
  • Auxiliary Data: The benchmark also provides 24 million unlabeled spectra and 118 million molecules (from PubChem) to support self-supervised learning and retrieval tasks.
Table 1: MassSpecGym is the largest publicly avail- able dataset of high-quality labeled MS/MS spec- tra. Our quality assessment workflow eliminates noisy or corrupted spectra and ensures reliable molecular labels and metadata (Section 3.3). The “Split” col- umn highlights that, unlike other large-scale datasets, MassSpecGym provides a pre-defined data split.

Key Insights and Ideas

  • Solving the Leakage Crisis: The authors demonstrate that standard splitting methods (based on 2D InChIKeys) allow "near-duplicates" to leak into the test set. Their MCES-based approach effectively eliminates this, providing a much more honest assessment of how a model will perform on truly novel molecules.
Figure 2: Our MCES-based data split resolves the data-leakage issue abun- dant in prior work. The standard ap- proach separates molecules with iden- tical planar structures (2D InChIKeys) into different folds, disregarding minor molecular modifications. This leads to near-duplicate test molecules (with Tan- imoto similarity > 0.85) being leaked from the training data (red), as shown in the example below. In contrast, our approach maximizes molecular edit dis- tance (MCES) between training and test sets, ensuring distinct data folds (green).
  • Metadata is Essential: The dataset is highly standardized, including critical metadata like collision energy and instrument type. This is vital because the same molecule can produce vastly different spectra under different experimental conditions.
  • Accessibility: By providing a user-friendly interface via PyTorch Lightning and Hugging Face, the authors have lowered the barrier to entry for ML researchers who may not have a background in analytical chemistry.

Key Results

The authors evaluated several baseline models (Transformers, GNNs, and Fingerprint-based models) and found that the problem is far from solved:

  • De Novo Generation: All baseline models achieved 0% exact match accuracy on the test set. This highlights the extreme difficulty of generating complex molecular graphs from noisy spectral data.
  • Retrieval: The state-of-the-art MIST model performed best but still only achieved a "Hit rate @ 1" of roughly 14.6%.
  • Spectrum Simulation: While models like FraGNNet showed promise, there remains a substantial gap between predicted and experimental spectra.

Limitations and Future Work

The authors acknowledge several limitations:

  • Ionization Mode: The current version focuses exclusively on positive ionization mode and singly-charged ions.
  • Noise: While they cleaned the data, complex noise sources like isotope signals and chimeric spectra (multiple molecules in one spectrum) remain challenging to handle perfectly.
  • Future Directions: The team plans to expand the benchmark to include molecular networking (clustering related spectra) and additional data modalities like Electron Ionization (EI) spectra.

MassSpecGym is publicly available at https://github.com/pluskal-lab/MassSpecGym.

Summary updated

Abstract

The discovery and identification of molecules in biological and environmental samples is crucial for advancing biomedical and chemical sciences. Tandem mass spectrometry (MS/MS) is the leading technique for high-throughput elucidation of molecular structures. However, decoding a molecular structure from its mass spectrum is exceptionally challenging, even when performed by human experts. As a result, the vast majority of acquired MS/MS spectra remain uninterpreted, thereby limiting our understanding of the underlying (bio)chemical processes. Despite decades of progress in machine learning applications for predicting molecular structures from MS/MS spectra, the development of new methods is severely hindered by the lack of standard datasets and evaluation protocols. To address this problem, we propose MassSpecGym -- the first comprehensive benchmark for the discovery and identification of molecules from MS/MS data. Our benchmark comprises the largest publicly available collection of high-quality labeled MS/MS spectra and defines three MS/MS annotation challenges: de novo molecular structure generation, molecule retrieval, and spectrum simulation. It includes new evaluation metrics and a generalization-demanding data split, therefore standardizing the MS/MS annotation tasks and rendering the problem accessible to the broad machine learning community. MassSpecGym is publicly available at https://github.com/pluskal-lab/MassSpecGym.
BibTeX
@inproceedings{bushuiev2024massspecgym,
  author = {Roman Bushuiev and Anton Bushuiev and Niek F. de Jonge and Adamo Young and Fleming Kretschmer and Raman Samusevich and Janne Heirman and Fei Wang and Luke Zhang and Kai Dührkop and Marcus Ludwig and Nils A. Haupt and Apurva Kalia and Corinna Brungs and Robin Schmid and Russell Greiner and Bo Wang and David S. Wishart and Li-Ping Liu and Juho Rousu and Wout Bittremieux and Hannes Rost and Tytus D. Mak and Soha Hassoun and Florian Huber and Justin J. J. van der Hooft and Michael A. Stravs and Sebastian Böcker and Josef Sivic and Tomáš Pluskal},
  title = {{MassSpecGym: A benchmark for the discovery and identification of molecules}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=AAo8zAShX3},
}
Figure