When should we trust the annotation? Selective prediction for molecular structure retrieval from mass spectra

Mira Jürgens, Gaetan De Waele, Morteza Rakhshaninejad, Willem Waegeman

TL;DR

Researchers have developed a selective prediction framework that allows machine learning models to abstain from identifying molecular structures when they are likely to be wrong. By evaluating uncertainty at the level of candidate rankings rather than just raw molecular bits, the team provides a way to guarantee a specific level of accuracy for high-stakes applications like clinical metabolomics.

What Problem Does This Paper Address?

In the field of metabolomics, scientists use tandem mass spectrometry (MS/MS) to identify the "dark matter" of the chemical world—the thousands of small molecules in biological samples that remain uncharacterized. While machine learning has gotten much better at matching these spectra to molecular structures, it still makes frequent mistakes.

In high-stakes environments—such as diagnosing a patient with a rare metabolic disorder or screening for environmental toxins—an incorrect "guess" by an AI can have serious real-world consequences. Currently, most models provide a "best guess" regardless of how sure they are. This paper addresses the need for a selective prediction framework: a way for the model to say, "I don't know," and only provide annotations when it meets a user-defined threshold of reliability.

Methods

The authors treat molecular identification as a two-stage retrieval problem: first, a model predicts a "molecular fingerprint" (a bit-vector of structural properties); second, it ranks candidate molecules from a database based on how well they match that fingerprint.

To decide when to trust a prediction, they evaluated several Uncertainty Quantification (UQ) strategies across two levels of granularity:

  1. Fingerprint-level: Measuring uncertainty over individual bits of the molecular structure.
  2. Retrieval-level: Measuring uncertainty over the final ranking of candidate molecules.

They compared standard "first-order" measures (like the confidence score of the top hit) against "second-order" Bayesian methods (using Deep Ensembles to see how much different versions of the model disagree). Crucially, they applied the Selection with Guaranteed Risk (SGR) algorithm, which uses statistical generalization bounds to ensure that the error rate on accepted predictions stays below a target level specified by the user.

Datasets

The study utilizes the MassSpecGym benchmark, a large-scale, standardized dataset for MS/MS retrieval.

  • Scale: 231,104 labeled spectra-molecule pairs.
  • Diversity: Covers 28,929 unique molecules.
  • Rigorous Splitting: The data is split into training and test sets based on molecular structure similarity (using Maximum Common Edge Subgraph distance). This ensures the model is tested on truly "unseen" chemical structures, rather than just new spectra of known molecules.

Key Insights and Ideas

  • The "Level Alignment" Principle: The most surprising finding was that uncertainty at the fingerprint level (bits) is a poor predictor of whether the final retrieval will be successful. A model might be very uncertain about a few bits of a molecule, but if those bits aren't critical for distinguishing it from other candidates in the database, the retrieval might still be easy. Conversely, a "confident" fingerprint prediction can fail if the database contains many nearly identical isomers.
  • Simple is Often Better: Computationally expensive Bayesian "epistemic" uncertainty (measuring what the model doesn't know) was consistently outperformed by simpler "total" uncertainty measures. For exact-match retrieval, the score gap (the distance between the #1 and #2 ranked candidates) was a remarkably effective trust signal.
  • Rank Variance for Top-K: When the goal is to provide a "top 5" or "top 20" list rather than a single answer, rank variance—how much the composition of that list changes when you perturb the model—emerged as the strongest criterion for selection.

Key Results

  • Provable Risk Control: The authors demonstrated that using the SGR algorithm, a practitioner can set a "tolerable error rate" (e.g., 10%) and the system will automatically filter the data to provide a subset of annotations that meet that guarantee with high probability.
  • Coverage Trade-offs: For a relaxed retrieval task (Hit@20), the framework could retain up to 87% of spectra while maintaining a controlled error rate. However, for exact-match retrieval (Hit@1), the base models are still struggling; the system could only guarantee high accuracy on a small, highly certain subset of the data.
  • Relative AURC: Using the "Area Under the Risk-Coverage curve" (AURC), they showed that retrieval-level scores significantly outperformed distance-based measures in the latent space, which are often used in other fields for out-of-distribution detection.

Limitations and Future Work

  • Model Architecture: The study used a standard Multi-Layer Perceptron (MLP) as the base. While effective, the authors note that testing this framework on more complex architectures, like Transformers, is a necessary next step.
  • Fingerprint Sparsity: Molecular fingerprints are "sparse" (mostly zeros). Current UQ methods don't always account for this extreme imbalance, which might be why bit-level uncertainty performed poorly.
  • Beyond Retrieval: The authors suggest that future work should explore "conformal prediction," which provides a set of possible candidates rather than just a single "accept/reject" decision, potentially offering even more flexibility for researchers.

Summary updated

Abstract

Machine learning methods for identifying molecular structures from tandem mass spectra (MS/MS) have advanced rapidly, yet current approaches still exhibit significant error rates. In high-stakes applications such as clinical metabolomics and environmental screening, incorrect annotations can have serious consequences, making it essential to determine when a prediction can be trusted. We introduce a selective prediction framework for molecular structure retrieval from MS/MS spectra, enabling models to abstain from predictions when uncertainty is too high. We formulate the problem within the risk-coverage tradeoff framework and comprehensively evaluate uncertainty quantification strategies at two levels of granularity: fingerprint-level uncertainty over predicted molecular fingerprint bits, and retrieval-level uncertainty over candidate rankings. We compare scoring functions including first-order confidence measures, aleatoric and epistemic uncertainty estimates from second-order distributions, as well as distance-based measures in the latent space. All experiments are conducted on the MassSpecGym benchmark. Our analysis reveals that while fingerprint-level uncertainty scores are poor proxies for retrieval success, computationally inexpensive first-order confidence measures and retrieval-level aleatoric uncertainty achieve strong risk-coverage tradeoffs across evaluation settings. We demonstrate that by applying distribution-free risk control via generalization bounds, practitioners can specify a tolerable error rate and obtain a subset of annotations satisfying that constraint with high probability.
BibTeX
@misc{jurgens2026when,
  author = {Mira Jürgens and Gaetan De Waele and Morteza Rakhshaninejad and Willem Waegeman},
  title = {{When should we trust the annotation? Selective prediction for molecular structure retrieval from mass spectra}},
  year = {2026},
  eprint = {2603.10950},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.10950},
}
Figure