FS-Mol: A Few-Shot Learning Dataset of Molecules

Megan Stanley, John F Bronskill, Krzysztof Maziarz, Hubert Misztela, Jessica Lanini, Marwin Segler, Nadine Schneider, Marc BrockschmidtNeurIPS 2021 · OpenReview

TL;DR

This paper introduces FS-Mol, a novel dataset and benchmark for few-shot learning in drug discovery, specifically for predicting molecular activity (QSAR). By providing a large collection of related, low-data tasks, FS-Mol aims to bridge the gap between current few-shot learning research (largely in computer vision) and the practical needs of drug discovery, where data is inherently scarce and expensive.

What Problem Does This Paper Address?

Drug discovery is a notoriously expensive and time-consuming process. A key step involves identifying and optimizing molecules that can interact with a specific biological target (e.g., a protein). This is often done by building Quantitative Structure-Activity Relationship (QSAR) models, which predict a molecule's activity based on its chemical structure. However, generating experimental data for QSAR is costly, leading to datasets with only a few dozen or hundreds of data points per target. While many such small, related datasets exist, current deep learning methods struggle with this "low-data" regime. Few-shot learning, which excels at adapting to new tasks with minimal data, has primarily been explored in computer vision. This paper addresses the need for a dedicated benchmark and dataset to advance few-shot learning techniques specifically for molecular property prediction in drug discovery.

Methods

The authors propose a comprehensive benchmarking framework for few-shot learning on molecular data. This involves:

  • Dataset Creation (FS-Mol): A large dataset of QSAR tasks is curated from the ChEMBL database. Each task represents a unique assay for a specific protein target. The dataset is split into a large pretraining set (Dtrain) and a disjoint set of few-shot testing tasks (Dtest).
  • Binary Classification: Activity data (e.g., IC50 values) is converted into a binary active/inactive classification problem, which is more robust to noise and class imbalance in real-world scenarios.
  • Evaluation Metric: Area Under the Precision-Recall Curve (AUPRC) is used as the primary metric, as it's sensitive to class imbalance. The metric is further refined to ∆AUPRC, which measures the improvement over a trivial baseline, addressing the research question of whether learned knowledge transfers gainfully to new tasks.
  • Benchmarking Procedure: A fixed procedure is defined for evaluating few-shot learning methods, including per-task dataset splitting (using stratified random sampling) and evaluation across various support set sizes to understand performance with limited data.
  • Baseline Models: The authors implement and evaluate several representative few-shot learning approaches, including:
  • Single-Task Models: Random Forests (RF) and k-Nearest Neighbors (kNN) on molecular features.
  • Multi-Task Pretraining: A Graph Neural Network (GNN) trained on multiple tasks simultaneously.
  • Self-Supervised Pretraining: A Molecule Attention Transformer (MAT) model.
  • Meta-Learning Models: GNN-MAML (optimization-based) and PN (metric-based).

Datasets

  • FS-Mol: The core contribution is the introduction of the FS-Mol dataset. It comprises 5120 assays (tasks) derived from ChEMBL27, involving 233,786 unique compounds. The dataset is split into:
  • Dtrain: 4938 tasks for pretraining.
  • Dtest: 157 tasks for few-shot evaluation.
  • Dvalid: 40 tasks for development. The dataset is designed to mimic real-world drug discovery challenges, with small task sizes (mean of 94 compounds per task) and a focus on drug-like molecules.

  • ChEMBL27: The raw data for FS-Mol is sourced from the ChEMBL database.

Key Insights and Ideas

  • Bridging the Gap: The paper highlights the critical need to move few-shot learning research beyond computer vision into scientific domains like drug discovery, where its potential impact is significant due to inherent data scarcity.
  • Task Similarity Matters: The results suggest that meta-learning approaches (like PN and GNN-MAML) show promise, especially when tasks are related, indicating that learning "how to learn" is beneficial.
  • Pretraining is Key: Pre-trained models (GNN-MT, GNN-MAML, PN) generally outperform models trained from scratch (GNN-ST), underscoring the importance of leveraging knowledge from related tasks.
  • Data Scarcity is a Challenge: While few-shot methods aim to work with limited data, the performance still generally improves with more available data in the support set, though some meta-learning models struggle to leverage this additional data effectively.
  • Real-World Relevance: The use of binary classification and metrics like ∆AUPRC makes the benchmark more aligned with practical drug discovery evaluation.

Key Results

  • PN Outperforms: The metric-based meta-learning model, PN, demonstrated the strongest performance, particularly at very small support set sizes (e.g., 16 compounds), significantly outperforming traditional methods like Random Forests.
  • Meta-learning Gains: GNN-MAML also showed gains over single-task baselines, especially with limited data.
  • Pretraining Benefits: Pre-trained models (GNN-MT, MAT) generally performed better than models trained from scratch (GNN-ST). However, MAT (self-supervised) underperformed GNN-MT and GNN-MAML (task-specific pretraining).
  • Task-Dependent Performance: While some methods showed clear advantages, the results also indicated that performance is highly task-dependent, and few-shot learning doesn't always guarantee improvement over well-tuned single-task models.
  • Support Set Size Impact: Performance generally increased with larger support set sizes, answering RQ2, though some meta-learning models showed less benefit from increased data.

Limitations and Future Work

  • Temporal Information: The dataset lacks inherent temporal information, necessitating the use of random splits for evaluation, which might not perfectly capture real-world temporal dynamics of data acquisition.
  • Proprietary Data: The paper acknowledges that proprietary industry data, which might be more recent and relevant than public ChEMBL data, could further improve pre-training.
  • Baseline Completeness: The authors note that the provided baselines are representative but not exhaustive, suggesting that variations of these methods or entirely new approaches could yield better results.
  • Transferability Understanding: Further research is needed to understand when and why transfer learning is effective for specific new tasks, as it's not always guaranteed.
  • Dataset Expansion: The authors mention that their extraction pipeline can be adapted to incorporate new data as it becomes available in future ChEMBL editions.

Summary updated

Abstract

Small datasets are ubiquitous in drug discovery as data generation is expensive and can be restricted for ethical reasons (e.g. in vivo experiments). A widely applied technique in early drug discovery to identify novel active molecules against a protein target is modelling quantitative structure-activity relationships (QSAR). It is known to be extremely challenging, as available measurements of compound activities range in the low dozens or hundreds. However, many such related datasets exist, each with a small number of datapoints, opening up the opportunity for few-shot learning after pre-training on a substantially larger corpus of data. At the same time, many few-shot learning methods are currently evaluated in the computer-vision domain. We propose that expansion into a new application, as well as the possibility to use explicitly graph-structured data, will drive exciting progress in few-shot learning. Here, we provide a few-shot learning dataset (FS-Mol) and complementary benchmarking procedure. We define a set of tasks on which few-shot learning methods can be evaluated, with a separate set of tasks for use in pre-training. In addition, we implement and evaluate a number of existing single-task, multi-task, and meta-learning approaches as baselines for the community. We hope that our dataset, support code release, and baselines will encourage future work on this extremely challenging new domain for few-shot learning.
BibTeX
@inproceedings{stanley2021fs,
  author = {Megan Stanley and John F Bronskill and Krzysztof Maziarz and Hubert Misztela and Jessica Lanini and Marwin Segler and Nadine Schneider and Marc Brockschmidt},
  title = {{FS-Mol: A Few-Shot Learning Dataset of Molecules}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=701FtuyLlAd},
}
Figure