A review of transformer models in drug discovery and beyond

Jian Jiang, Long Chen, Ke Lü, Bozheng Dou, Chunhuan Zhang, Hongsong Feng, Yueying Zhu, Huahai Qiu, Bengong Zhang, Guo‐Wei WeiJournal of Pharmaceutical Analysis 2024 · DOI

TL;DR

Transformer models, originally from natural language processing, are revolutionizing drug discovery by effectively handling complex biological and chemical data. This review highlights their diverse applications, from designing new proteins and predicting molecular dynamics to identifying drug targets and optimizing drug leads, showcasing their immense potential to accelerate the development of new medicines.

What Problem Does This Paper Address?

The drug discovery process is notoriously complex, time-consuming, and expensive. Traditional methods often struggle with the vast and intricate datasets involved, such as understanding long-range dependencies in molecular structures or analyzing high-dimensional biological data. This paper addresses the need for more efficient and powerful computational tools to navigate these challenges. It specifically focuses on how transformer models, with their unique ability to capture contextual information and long-range dependencies, are being applied across various stages of drug discovery and related scientific fields, bridging gaps between disciplines like biology, chemistry, and computer science.

Methods

The paper reviews the application of transformer models, a deep learning architecture that relies on the self-attention mechanism. This mechanism allows the model to weigh the importance of different parts of the input data, enabling it to understand complex relationships within sequences. The review covers various transformer-based approaches, including:

  • Encoder-Decoder Architectures: The foundational structure of transformers, used for sequence-to-sequence tasks.
  • Self-Attention Mechanisms: The core innovation that allows transformers to process data in parallel and capture long-range dependencies.
  • Multi-Head Attention: An extension that enables the model to attend to different aspects of the data simultaneously.
  • Pre-trained Models: Leveraging large datasets to train models that can then be fine-tuned for specific tasks, significantly reducing the need for extensive task-specific data.
  • Domain-Specific Adaptations: Transformers are being adapted with specialized embeddings and architectures (e.g., for molecular structures, chemical images, and biological sequences) to excel in their respective domains.
  • Integration with Other Techniques: Transformers are combined with methods like topological data analysis, graph neural networks, and molecular dynamics simulations to enhance their capabilities.

Datasets

The paper discusses the use of various datasets across different applications, including:

  • Protein Data Bank (PDB): Used for protein structure analysis.
  • PubChem: A large database of chemical compounds, used for molecular property prediction and image analysis.
  • DrugBank: A comprehensive database of drugs, used for virtual screening and drug repurposing.
  • PDBbind-2016 core set: A benchmark dataset for protein-ligand binding affinity prediction.
  • Tox21 and SIDER datasets: Used for predicting chemical toxicity and adverse side effects.
  • scRNA-seq data: Single-cell RNA sequencing data from various human and mouse tissues for cell-type annotation.
  • Various reaction and molecular property datasets: Used for training models in chemical synthesis and prediction.

The paper highlights that while large datasets are beneficial, transformers also show promise in handling "small data set challenges" through techniques like transfer learning and data augmentation.

Key Insights and Ideas

  • Beyond NLP: Transformers are not just for text; their ability to understand sequential and relational data makes them incredibly versatile for complex biological and chemical sequences and structures.
  • Long-Range Dependencies are Key: The self-attention mechanism is crucial for understanding how distant parts of a molecule or protein interact, a capability that traditional models often lacked.
  • Data-Centric Revolution: Transformers are driving a shift towards data-driven approaches in drug discovery, enabling more accurate predictions and faster exploration of chemical space.
  • Interdisciplinary Bridge: These models are effectively bridging the gap between biology, chemistry, and computer science, fostering collaboration and innovation.
  • "Chemical Language" Understanding: Transformers are learning to interpret chemical representations (like SMILES strings) and even chemical images, opening new avenues for analysis and design.
  • Small Data, Big Impact: Even with limited data, pre-trained transformers combined with transfer learning can yield surprisingly good results, making them valuable for niche problems.

Key Results

The paper reviews numerous applications where transformer models have demonstrated significant improvements:

  • Protein Engineering: Models like TopFit and MSA transformer have reduced costs and accelerated protein engineering by accurately predicting protein fitness landscapes and mutation effects.
  • Virtual Screening (VS): Transformer-enabled sequence-based VS (SVS) models have shown superior performance in predicting protein-ligand binding affinity compared to traditional structure-based methods, achieving success rates of 72% for low-level measurement and 63% for high-level measurement on CASF-2007.
  • Chemical Image Analysis: Models like DECIMER have achieved remarkable accuracy (up to 96%) in predicting SMILES strings from chemical images.
  • Drug Target Identification: Novel transformer architectures like TopoFormer, built on 3D molecular structures, have outperformed others in scoring tasks, achieving success rates of 72% and 63% on CASF-2007 for ranking and docking tasks, respectively.
  • Small Data Challenges: Models integrating transformers have shown superior performance on datasets with as little as 1% labeled data, outperforming other methods in predicting target attributes.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Interpretability: While powerful, understanding why a transformer makes a certain prediction remains a challenge, especially in complex biological systems.
  • Data Scarcity: For certain drug discovery tasks, the availability of high-quality, experimentally validated data is limited, which can hinder model training and generalization.
  • Computational Cost: Training large transformer models can be computationally intensive, requiring significant resources.
  • Generalization: Ensuring models generalize well to new, unseen data and chemical spaces is an ongoing area of research.
  • Ethical Considerations: Particularly in areas like drug addiction, ethical and safety aspects of AI-driven solutions need careful consideration.

Future work is expected to focus on:

  • Large Language Models (LLMs): Exploring the role of LLMs like ChatGPT in hypothesis generation, molecular design, and clinical trial management.
  • Multimodal Learning: Integrating diverse omics data for a more holistic understanding of biological systems.
  • Customization: Developing tailored transformer models for specific drug discovery challenges.
  • Enhanced Interpretability: Developing methods to better understand the decision-making process of transformer models.
  • Addressing Data Sparsity: Continued development of techniques for effective learning from limited data.

Summary updated

Abstract

Transformer models have emerged as pivotal tools within the realm of drug discovery, distinguished by their unique architectural features and exceptional performance in managing intricate data landscapes. Leveraging the innate capabilities of transformer architectures to comprehend intricate hierarchical dependencies inherent in sequential data, these models showcase remarkable efficacy across various tasks, including new drug design and drug target identification. The adaptability of pre-trained transformer-based models renders them indispensable assets for driving data-centric advancements in drug discovery, chemistry, and biology, furnishing a robust framework that expedites innovation and discovery within these domains. Beyond their technical prowess, the success of transformer-based models in drug discovery, chemistry, and biology extends to their interdisciplinary potential, seamlessly combining biological, physical, chemical, and pharmacological insights to bridge gaps across diverse disciplines. This integrative approach not only enhances the depth and breadth of research endeavors but also fosters synergistic collaborations and exchange of ideas among disparate fields. In our review, we elucidate the myriad applications of transformers in drug discovery, as well as chemistry and biology, spanning from protein design and protein engineering, to molecular dynamics (MD), drug target identification, transformer-enabled drug virtual screening (VS), drug lead optimization, drug addiction, small data set challenges, chemical and biological image analysis, chemical language understanding, and single cell data. Finally, we conclude the survey by deliberating on promising trends in transformer models within the context of drug discovery and other sciences.
BibTeX
@article{jiang2024review,
  author = {Jian Jiang and Long Chen and Ke Lü and Bozheng Dou and Chunhuan Zhang and Hongsong Feng and Yueying Zhu and Huahai Qiu and Bengong Zhang and Guo‐Wei Wei},
  title = {{A review of transformer models in drug discovery and beyond}},
  journal = {Journal of Pharmaceutical Analysis},
  year = {2024},
  doi = {10.1016/j.jpha.2024.101081},
}
Figure