In the race to discover new molecules and materials, machine learning (ML) has become an indispensable tool. However, training these models often requires massive datasets labeled with high-fidelity simulations (like Density Functional Theory), which are computationally expensive. A new paper by Nguyen et al. systematically explores how we can work smarter, not harder, by using Active Learning (AL) to curate training data more efficiently.
TL;DR
This study benchmarks different active learning strategies across molecular and materials datasets, revealing that uncertainty-based sampling is the most robust way to reduce labeling costs. The authors find that many popular datasets are highly redundant, and by picking the right samples, researchers can achieve near-peak model performance using as little as 13–25% of the available data.
What Problem Does This Paper Address?
Most ML models in chemistry and materials science are trained on datasets curated through "passive" or random sampling. This often leads to a "Goldilocks" problem: some regions of chemical space are over-represented (redundancy), while critical, high-information regions are ignored.
Because generating labels (e.g., calculating a molecule's energy) is the primary bottleneck, we need a way to identify which specific samples will provide the most "knowledge" to the model. This is the core of Active Learning: an iterative process where the model "asks" for the labels of samples it finds most informative. While AL is a known concept, its effectiveness across different model architectures (classical vs. deep learning) and different types of data (molecules vs. crystals) hasn't been systematically compared—until now.
Methods
The researchers compared three primary data acquisition strategies:
- Random Sampling (RS): The baseline "passive" approach.
- Uncertainty Sampling (US): Using a "query-by-committee" approach. The system picks samples where a committee of models disagrees the most on the predicted value.
- Diversity Sampling (DS): A geometric approach (using the TypiClust algorithm) that picks samples to ensure the training set covers the widest possible variety of structures in the feature space.
They tested these strategies using two distinct model classes:
- Descriptor-based Ensembles: A combination of Random Forest, Kernel Ridge Regression, and XGBoost using traditional chemical descriptors (like fingerprints).
- Deep Learning: SchNet, a continuous-filter convolutional neural network that learns representations directly from 3D atomic coordinates.
Datasets
The study utilized four datasets to capture different scales and domains:
- QM9 & QM9small: ~134k small organic molecules (and a 5% subset) for predicting the HOMO–LUMO gap.
- Matbench Materials Project (MP): ~133k inorganic crystals for predicting formation energy.
- Pourbaix Stability: A smaller, high-stakes dataset of ~6.8k crystals with expensive HSE06-level labels.
Key Insights and Ideas
- Massive Redundancy: The most striking "aha moment" is the sheer amount of filler in standard datasets. For the large QM9 and Matbench MP datasets, only 15–25% of the data was needed to reach within 10% of the "full-data" accuracy.
- Uncertainty Trumps Diversity: In almost every scenario, Uncertainty Sampling (US) outperformed Diversity Sampling (DS). This suggests that model disagreement is a much more potent signal for informativeness than structural variety alone.
- Model Choice > Sampling Strategy: While AL helps, the choice of model architecture is often more impactful. SchNet (the deep learning model) consistently outperformed the classical ensemble, even when the ensemble was given significantly more data.
- The "Data-Scarce" Sweet Spot: AL shows its greatest value when data is truly limited and expensive. On the Pourbaix dataset, Uncertainty Sampling saved roughly 1,200 labels (22% of the pool) compared to random sampling to reach the same accuracy threshold.
Key Results
The authors quantified data efficiency using $N_{10\%}$—the number of samples required to reach a performance level within 10% of the model trained on the entire dataset.
- Efficiency Gains: For the descriptor-based ensemble on QM9, Uncertainty Sampling reached the threshold at 19,000 samples (17% of the pool), whereas Random Sampling required 26,000 (25%).
- Deep Learning Superiority: SchNet achieved an RMSE of 0.090 eV on QM9, vastly outperforming the ensemble's 0.339 eV. As shown in Figure 6, SchNet is superior at every budget level, though it requires more data to reach its own (much lower) "full-data" error threshold.
- Dataset Specificity: While US was generally best, the "best" strategy for SchNet actually shifted depending on the dataset (US for molecules, DS for crystals), highlighting that there is no "one-size-fits-all" recipe for deep learning.
Limitations and Future Work
The authors noted several constraints:
- Computational Cost: Due to GPU limits, SchNet models were trained using only a single run per configuration, meaning the statistical variance (error bars) for the deep learning curves could not be rigorously assessed.
- Limited Strategy Suite: The study focused on three core strategies. Future work could explore hybrid approaches that combine uncertainty and diversity.
- Benchmarking Platform: The authors aim to develop a broader benchmarking platform to help the community quantify redundancy in new target properties and select the best AL strategies for their specific budgets.
For researchers in the field, the takeaway is clear: if you have a limited labeling budget, start with uncertainty-based sampling and, if structural data is available, prioritize deep learning architectures like SchNet over classical ensembles.
Summary updated