Abstract
Regression tasks are essential in many fields, including chemistry, where property prediction models are used to prioritize chemical compounds for experimental testing. In this context, it is common to maximize properties, such as potency, which measures the extent to which substances achieve their intended biological effect. Conventional evaluation metrics and loss functions used to train prediction models for these tasks prioritize average performance, assuming that all domain values are equally relevant. However, this does not correspond to real-world expectations. In this paper, we argue the urgent need to reassess current evaluation and optimization practices in drug discovery with implications for tasks beyond the chemistry domain where non-uniform domain preferences are observed. Here, we use data on ten potency classes to compare model outcomes optimized and selected using traditional loss functions and distribution-specific methods: a feature space design method and a recent function that accounts for non-uniform domain preferences. Our empirical results show that models using the latter methods identify more unique and better-performing compounds when compared to models optimized with traditional tools. While identifying relevant compounds detected by other models, results show that accounting for non-uniform domain preferences enhances predictive performance in the most relevant cases and more effectively distinguishes between less and more relevant instances. Critically, our results underscore the importance of reevaluating the optimization and evaluation methods used in critical domains such as chemistry and their impact on other natural and physical domains.
Introduction: The "Average Performance Trap"
In compound potency prediction, regression models estimate a molecule's biological activity (often measured as $\text{pIC}_{50}$). The ultimate goal is to identify "hits"—rare compounds with exceptionally high potency. However, a fundamental disconnect exists between model training and real-world application. Most datasets are imbalanced, containing thousands of mediocre compounds and very few highly potent ones. Standard loss functions, such as Mean Squared Error (MSE) and Mean Absolute Error (MAE), reward models for minimizing total error across the entire dataset. Consequently, models prioritize accuracy in the low-to-medium potency range where data is abundant, often failing to identify the most promising drug candidates in the high-potency range.
Methodology: Distribution-Specific Optimization
To address this, we compared traditional optimization approaches against two "distribution-specific" methods designed for non-uniform domain preferences:
- Squared Error-Relevance Area (SERA): This method computes a relevance function mapping $\text{pIC}_{50}$ values to a 0-1 scale, often using Piecewise Cubic Hermite Interpolating Polynomials. It calculates the area under the curve of squared errors for subsets of cases with increasing relevance, forcing the model to prioritize accuracy in high-relevance regions.
- Autofocused Oracle (AFO): An adaptive retraining method that iteratively updates the predictive model to focus on the most promising regions of the design space. It calculates importance weights for samples based on their likelihood of exceeding a performance threshold.
These methods were tested across five common architectures: k-Nearest Neighbors (kNN), Support Vector Regression (SVR), XGBoost (XGB), Deep Neural Networks (DNN), and Graph Convolutional Neural Networks (GCN).
Experimental Setup
The study utilized a dataset derived from ChEMBL (version 30), consisting of 10 activity classes with at least 1,000 compounds each, totaling 13,444 qualifying compounds. Potency values ($\text{pIC}_{50}$) ranged from 5 to 11, with the "relevant" high-potency range defined as 9–11.
Key Results and Discussion
Our empirical results demonstrate that relevance-weighted models do not just find more potent compounds; they find different ones. Optimization with SERA and AFO allowed models to explore regions of the chemical space that traditional MSE-optimized models ignored.
As shown in Figure 4, models optimized with SERA consistently identified a higher number of true high-potency compounds ($\text{pIC}_{50} \ge 9$) compared to those using MAE or MSE. Notably, $GCN_{AFO}$ and $DNN_{SERA}$ were particularly effective at identifying unique high-potency compounds—those not captured by any other metric/algorithm combination. While traditional models often had lower error in the low-potency range, SERA-optimized models maintained lower errors as relevance increased, effectively distinguishing between "good" and "great" instances.
Conclusion
The "average performance trap" in standard metrics can hide the failure of models to perform in the most critical regions of the domain. By explicitly defining relevance through methods like SERA and AFO, researchers can significantly improve the discovery of novel, high-potency compounds. This rethinking of evaluation should extend beyond chemistry to other fields with non-uniform preferences, such as meteorology or environmental science, where predicting rare, extreme events is paramount.
Summary updated