Structured Multi-task Learning for Molecular Property Prediction

Shengchao Liu, Meng Qu, Zuobai Zhang, Huiyu Cai, Jian Tang

Abstract

Multi-task learning for molecular property prediction is becoming increasingly important in drug discovery. However, in contrast to other domains, the performance of multi-task learning in drug discovery is still not satisfying as the number of labeled data for each task is too limited, which calls for additional data to complement the data scarcity. In this paper, we study multi-task learning for molecular property prediction in a novel setting, where a relation graph between tasks is available. We first construct a dataset including around 400 tasks as well as a task relation graph. Then to better utilize such relation graph, we propose a method called SGNN-EBM to systematically investigate the structured task modeling from two perspectives. (1) In the latent space, we model the task representations by applying a state graph neural network (SGNN) on the relation graph. (2) In the output space, we employ structured prediction with the energy-based model (EBM), which can be efficiently trained through noise-contrastive estimation (NCE) approach. Empirical results justify the effectiveness of SGNN-EBM.

Introduction

Predicting molecular properties, such as toxicity or binding affinity, is a foundational challenge in drug discovery. However, experimental data is often scarce due to the high cost and time required for collection. While Multi-Task Learning (MTL) aims to mitigate this "data scarcity" by sharing knowledge across related tasks, traditional MTL methods often treat tasks as independent entities or attempt to infer relationships solely from training gradients. This approach ignores significant biological domain knowledge, such as protein-protein interaction (PPI) networks.

This research introduces SGNN-EBM, a framework designed to leverage explicit task relation graphs. By modeling task correlations in both the latent feature space and the final output space, the model effectively utilizes biological priors to improve prediction accuracy.

Methods: The SGNN-EBM Framework

The SGNN-EBM architecture integrates two core components to capture task structures:

The SGNN-EBM architecture overview, showing the transition from input molecular and task graphs to latent space modeling and final energy-based output prediction.

1. Latent Space Modeling (SGNN)

The model employs a State Graph Neural Network (SGNN) that operates on the task relation graph. Unlike standard GNNs, the SGNN learns task representations by passing messages between related nodes while maintaining distinct "states" for each node and edge. These states correspond to potential labels (e.g., active vs. inactive), allowing the model to learn how the state of one biological target influences its neighbors in the latent space.

2. Output Space Modeling (EBM)

To capture dependencies between final labels, the framework uses an Energy-Based Model (EBM). Instead of predicting each task independently via simple sigmoid outputs, the EBM defines a joint "energy" for the entire configuration of labels. Lower energy states represent more probable label sets, enabling the model to learn statistical co-occurrences of properties.

3. Efficient Training via NCE

Training EBMs is computationally intensive due to the intractable partition function. The authors utilize Noise Contrastive Estimation (NCE) with an "energy tilting" strategy. A pre-trained backbone model serves as the noise distribution, focusing the training process on the most challenging examples to improve discriminative power.

Datasets and Benchmarking

To evaluate the framework, the authors merged the ChEMBL database (bioactive molecules) with the STRING database (protein-protein interactions). This resulted in a comprehensive Task Relation Graph where edges represent biological interactions. Three benchmark variants—ChEMBL 10, 50, and 100—were created based on different data sparsity thresholds.

Statistics for the constructed ChEMBL datasets at different sparsity thresholds.

Key Results and Insights

Empirical evaluations using the mean ROC-AUC across all tasks demonstrate the superiority of SGNN-EBM:

  • Performance Gains: SGNN-EBM consistently outperformed both Single-Task Learning (STL) and standard MTL baselines. On the ChEMBL 10 dataset, SGNN-EBM achieved an ROC-AUC of 78.35%, significantly higher than STL (71.67%) and standard MTL (74.83%).
  • Ablation Study: Results indicate that while the SGNN component provides the primary performance boost, the EBM-based output modeling adds a consistent 1-2% improvement, confirming the value of modeling label dependencies.
  • Biological Priors: The success of the model highlights that biological interaction data serves as a powerful prior. If two proteins are known to interact, a molecule's effect on one is a strong predictor for the other.

Conclusion and Future Work

SGNN-EBM represents a significant step toward integrating domain knowledge into molecular property prediction. While the model achieves state-of-the-art results, it faces limitations regarding inference speed due to the iterative nature of Gibbs sampling required by the EBM. Future research directions include optimizing inference efficiency, exploring higher-order dependencies, and automatically extracting task graphs in domains where explicit biological networks are unavailable.

Summary updated

Abstract

Multi-task learning for molecular property prediction is becoming increasingly important in drug discovery. However, in contrast to other domains, the performance of multi-task learning in drug discovery is still not satisfying as the number of labeled data for each task is too limited, which calls for additional data to complement the data scarcity. In this paper, we study multi-task learning for molecular property prediction in a novel setting, where a relation graph between tasks is available. We first construct a dataset including around 400 tasks as well as a task relation graph. Then to better utilize such relation graph, we propose a method called SGNN-EBM to systematically investigate the structured task modeling from two perspectives. (1) In the latent space, we model the task representations by applying a state graph neural network (SGNN) on the relation graph. (2) In the output space, we employ structured prediction with the energy-based model (EBM), which can be efficiently trained through noise-contrastive estimation (NCE) approach. Empirical results justify the effectiveness of SGNN-EBM. Code is available on https://github.com/chao1224/SGNN-EBM.
BibTeX
@inproceedings{liu2022structured,
  author = {Shengchao Liu and Meng Qu and Zuobai Zhang and Huiyu Cai and Jian Tang},
  title = {{Structured Multi-task Learning for Molecular Property Prediction}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2022},
  url = {https://openreview.net/forum?id=ez7505m9gLv},
}
Figure