TL;DR
This paper introduces FS-Mol, a novel dataset and benchmark for few-shot learning in drug discovery, specifically for predicting molecular activity (QSAR). By providing a large collection of related, low-data tasks, FS-Mol aims to bridge the gap between current few-shot learning research (largely in computer vision) and the practical needs of drug discovery, where data is inherently scarce and expensive.
What Problem Does This Paper Address?
Drug discovery is a notoriously expensive and time-consuming process. A key step involves identifying and optimizing molecules that can interact with a specific biological target (e.g., a protein). This is often done by building Quantitative Structure-Activity Relationship (QSAR) models, which predict a molecule's activity based on its chemical structure. However, generating experimental data for QSAR is costly, leading to datasets with only a few dozen or hundreds of data points per target. While many such small, related datasets exist, current deep learning methods struggle with this "low-data" regime. Few-shot learning, which excels at adapting to new tasks with minimal data, has primarily been explored in computer vision. This paper addresses the need for a dedicated benchmark and dataset to advance few-shot learning techniques specifically for molecular property prediction in drug discovery.
Methods
The authors propose a comprehensive benchmarking framework for few-shot learning on molecular data. This involves:
- Dataset Creation (FS-Mol): A large dataset of QSAR tasks is curated from the ChEMBL database. Each task represents a unique assay for a specific protein target. The dataset is split into a large pretraining set (Dtrain) and a disjoint set of few-shot testing tasks (Dtest).
- Binary Classification: Activity data (e.g., IC50 values) is converted into a binary active/inactive classification problem, which is more robust to noise and class imbalance in real-world scenarios.
- Evaluation Metric: Area Under the Precision-Recall Curve (AUPRC) is used as the primary metric, as it's sensitive to class imbalance. The metric is further refined to ∆AUPRC, which measures the improvement over a trivial baseline, addressing the research question of whether learned knowledge transfers gainfully to new tasks.
- Benchmarking Procedure: A fixed procedure is defined for evaluating few-shot learning methods, including per-task dataset splitting (using stratified random sampling) and evaluation across various support set sizes to understand performance with limited data.
- Baseline Models: The authors implement and evaluate several representative few-shot learning approaches, including:
- Single-Task Models: Random Forests (RF) and k-Nearest Neighbors (kNN) on molecular features.
- Multi-Task Pretraining: A Graph Neural Network (GNN) trained on multiple tasks simultaneously.
- Self-Supervised Pretraining: A Molecule Attention Transformer (MAT) model.
- Meta-Learning Models: GNN-MAML (optimization-based) and PN (metric-based).
Datasets
- FS-Mol: The core contribution is the introduction of the FS-Mol dataset. It comprises 5120 assays (tasks) derived from ChEMBL27, involving 233,786 unique compounds. The dataset is split into:
- Dtrain: 4938 tasks for pretraining.
- Dtest: 157 tasks for few-shot evaluation.
-
Dvalid: 40 tasks for development. The dataset is designed to mimic real-world drug discovery challenges, with small task sizes (mean of 94 compounds per task) and a focus on drug-like molecules.
-
ChEMBL27: The raw data for FS-Mol is sourced from the ChEMBL database.
Key Insights and Ideas
- Bridging the Gap: The paper highlights the critical need to move few-shot learning research beyond computer vision into scientific domains like drug discovery, where its potential impact is significant due to inherent data scarcity.
- Task Similarity Matters: The results suggest that meta-learning approaches (like PN and GNN-MAML) show promise, especially when tasks are related, indicating that learning "how to learn" is beneficial.
- Pretraining is Key: Pre-trained models (GNN-MT, GNN-MAML, PN) generally outperform models trained from scratch (GNN-ST), underscoring the importance of leveraging knowledge from related tasks.
- Data Scarcity is a Challenge: While few-shot methods aim to work with limited data, the performance still generally improves with more available data in the support set, though some meta-learning models struggle to leverage this additional data effectively.
- Real-World Relevance: The use of binary classification and metrics like ∆AUPRC makes the benchmark more aligned with practical drug discovery evaluation.
Key Results
- PN Outperforms: The metric-based meta-learning model, PN, demonstrated the strongest performance, particularly at very small support set sizes (e.g., 16 compounds), significantly outperforming traditional methods like Random Forests.
- Meta-learning Gains: GNN-MAML also showed gains over single-task baselines, especially with limited data.
- Pretraining Benefits: Pre-trained models (GNN-MT, MAT) generally performed better than models trained from scratch (GNN-ST). However, MAT (self-supervised) underperformed GNN-MT and GNN-MAML (task-specific pretraining).
- Task-Dependent Performance: While some methods showed clear advantages, the results also indicated that performance is highly task-dependent, and few-shot learning doesn't always guarantee improvement over well-tuned single-task models.
- Support Set Size Impact: Performance generally increased with larger support set sizes, answering RQ2, though some meta-learning models showed less benefit from increased data.
Limitations and Future Work
- Temporal Information: The dataset lacks inherent temporal information, necessitating the use of random splits for evaluation, which might not perfectly capture real-world temporal dynamics of data acquisition.
- Proprietary Data: The paper acknowledges that proprietary industry data, which might be more recent and relevant than public ChEMBL data, could further improve pre-training.
- Baseline Completeness: The authors note that the provided baselines are representative but not exhaustive, suggesting that variations of these methods or entirely new approaches could yield better results.
- Transferability Understanding: Further research is needed to understand when and why transfer learning is effective for specific new tasks, as it's not always guaranteed.
- Dataset Expansion: The authors mention that their extraction pipeline can be adapted to incorporate new data as it becomes available in future ChEMBL editions.
Summary updated