Multitask finetuning and acceleration of chemical pretrained models for small molecule drug property prediction

Matthew Adrian, Yunsie Chung, Kevin Boyd, Saee Paliwal, Srimukh Prasad Veccham, Alan C. ChengarXiv 2025 · DOI · arXiv

Abstract

Predicting Absorption, Distribution, Metabolism, Excretion, and Toxicity (ADMET) properties is a cornerstone of drug discovery, yet it is often hampered by limited experimental data. While chemical foundation models pretrained on massive unlabeled datasets offer a solution, their standard single-task finetuning often leaves performance on the table. This study introduces KERMT (Kinetic GROVER Multi-Task), an optimized architecture that leverages multitask finetuning and GPU acceleration. We demonstrate that multitask learning acts as a robust regularizer and, significantly, that the benefits of pretraining scale positively with dataset size, challenging the common assumption that foundation models are primarily for low-data regimes.

Introduction

In the pharmaceutical industry, the ability to accurately predict a molecule's ADMET profile and biological potency is critical for reducing the high attrition rates of drug candidates. Deep learning models, particularly Graph Neural Networks (GNNs), have shown promise but frequently struggle with the "small data" problem inherent in early-stage discovery.

Chemical foundation models attempt to mitigate this by learning general chemical representations from millions of unlabeled molecules. However, the transition from general pretraining to specific property prediction often ignores the inherent relationships between different ADMET endpoints. This research investigates whether simultaneous multitask finetuning can improve predictive accuracy and how these models behave when applied to industrial-scale datasets.

Methodology and Model Architecture

The core of this work is KERMT, an enhanced iteration of the GROVER architecture. Unlike traditional approaches that finetune a separate model for each property, KERMT utilizes a shared encoder to generate molecular embeddings, which are then processed by multiple task-specific "heads" for simultaneous prediction.

Overview of the pretraining and finetuning workflow, alongside the specific pretraining algorithms (MoLFormer, KPGT, and KERMT) evaluated in this study.

To make large-scale training feasible, the authors implemented several technical optimizations:

  • Distributed Training: KERMT was reimplemented using PyTorch Distributed Data Parallel (DDP) and integrated with automated hyperparameter optimization.
  • Data Pipeline Acceleration: By using the cuik-molmaker package, molecular featurization was moved directly into the data-loading pipeline, eliminating CPU bottlenecks and reducing memory usage by 34%.
  • Comparative Benchmarking: KERMT was compared against state-of-the-art models including KPGT (a graph transformer) and MoLFormer (a language-based transformer).

Results and Discussion

The Scaling Surprise

A pivotal finding of this study is the relationship between dataset size and the efficacy of pretraining. While it is commonly assumed that foundation models provide the most value when data is scarce, the authors observed that KERMT’s performance advantage over non-pretrained baselines actually increased as the training set grew (specifically above 60,000 samples). This suggests that pretrained models possess a higher capacity or "performance ceiling" that is only fully realized with sufficient finetuning data.

Comparison of model performance (N_best count) across different training set sizes, highlighting KERMT's dominance as data volume increases.

Graph-Based vs. Language-Based Representations

The results consistently favored graph-based models (KERMT, KPGT) over the language-based MoLFormer. This indicates that explicitly modeling the 2D connectivity of atoms provides a more informative prior for ADMET properties than treating SMILES strings as linear sequences.

Multitask Learning as a Regularizer

Multitask finetuning proved to be a powerful regularizer. By learning multiple related properties at once, KERMT developed more generalizable representations. On the internal Merck data, KERMT MT statistically outperformed other models in 18 out of 30 assays. Furthermore, the optimized implementation achieved a 2.2x speedup in finetuning and a 2.9x speedup in inference compared to the original GROVER baseline.

Conclusion and Future Directions

This study demonstrates that multitask finetuning of chemical foundation models significantly enhances drug property prediction, particularly at scale. However, limitations remain; for instance, the models showed no significant improvement for cyclic peptides, likely due to a lack of such structures in the pretraining data. Future work will focus on specialized pretraining strategies for larger molecular modalities and exploring "partial freezing" techniques to further optimize the finetuning process.

Summary updated

BibTeX
@misc{adrian2025multitask,
  author = {Matthew Adrian and Yunsie Chung and Kevin Boyd and Saee Paliwal and Srimukh Prasad Veccham and Alan C. Cheng},
  title = {{Multitask finetuning and acceleration of chemical pretrained models for small molecule drug property prediction}},
  year = {2025},
  eprint = {2510.12719},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.12719},
}
Figure