Unified Deep Learning of Molecular and Protein Language Representations with T5ProtChem

Thomas Kelly, Song Xia, Jieyu Lu, Yingkai ZhangJournal of Chemical Information and Modeling 2025 · DOI

TL;DR

This paper introduces T5ProtChem, a unified deep learning model that can simultaneously process both molecular and protein sequences. By leveraging a novel "ProtiSMILES" pretraining objective, T5ProtChem bridges the gap between these domains, achieving state-of-the-art results in tasks like binding affinity and reaction prediction, and showing promise for protein engineering and drug discovery.

What Problem Does This Paper Address?

Traditional deep learning approaches often treat molecular and protein sequences as separate entities, relying on distinct models or concatenated architectures. This separation fails to capitalize on the shared atomic foundations underlying both molecular and protein structures. While existing models excel within their respective domains (e.g., AlphaFold for proteins, ChemBERTa for molecules), there's a growing need for unified models that can efficiently and effectively handle both, especially for interdisciplinary tasks like drug discovery where interactions between molecules and proteins are central. The challenge lies in creating a single model that can learn from and generalize across these distinct but related data types.

Methods

T5ProtChem is built upon the T5 (Text-to-Text Transfer Transformer) architecture, a powerful encoder-decoder model. The key innovation is the ProtiSMILES pretraining objective. This objective adapts the span-masking strategy from T5:

  • Span-Masking: Similar to T5, T5ProtChem masks contiguous spans of tokens in the input sequence.
  • ProtiSMILES Objective: For protein sequences, masked amino acid spans are replaced with sentinel tokens. The decoder is then tasked with generating the corresponding SMILES (Simplified Molecular Input Line Entry System) sequence for those masked amino acids. This effectively trains the model to "translate" between protein and chemical representations.
  • Unified Tokenization: To distinguish between amino acid and chemical tokens, amino acids are prepended with a special <P> token (e.g., <P>S).
  • Character-Level Tokenization: The model operates at the character level for both protein and chemical sequences.

This unified approach allows T5ProtChem to be fine-tuned for a variety of tasks, including protein function prediction, forward reaction prediction, binding affinity estimation, and novel tasks like covalent binder classification and sequence-level adduct prediction.

Datasets

  • Pretraining:
    • Proteins: UniRef50 (similar to ESM-2's pretraining data) and a subset of the AlphaFold DB Swiss-Prot data (AF-500 K) for validation.
  • Chemicals: PubChem (a large dataset of SMILES sequences).
  • Fine-tuning:
    • Forward Reaction Prediction: Used the same dataset as T5Chem.
  • Protein Function Prediction: DeepFRI dataset (for Gene Ontology terms).
  • Binding Affinity Prediction: Refined BindingDB dataset.
  • Covalent Binder Tasks: A newly curated dataset derived from CovBinderInPDB and CovalentInDB.

Key Insights and Ideas

  • Bridging the Domain Gap: The ProtiSMILES objective is a clever way to force the model to learn the underlying atomic relationships between protein and chemical sequences, moving beyond simple concatenation.
  • Unified Representation is Powerful: A single model trained on both domains can outperform specialized models or combined models on tasks that inherently involve interactions between proteins and molecules.
  • Novel Tasks Enabled: The unified approach opens doors to new applications, such as predicting covalent adducts, which require understanding both protein modification sites and chemical structures.
  • Efficiency and Generalizability: By learning a shared representation, T5ProtChem is more efficient and generalizable, reducing the need for multiple domain-specific models.

Key Results

  • Forward Reaction Prediction: Achieved state-of-the-art performance, surpassing established baselines like molecular transformers and T5Chem.
  • Protein Function Prediction: Showed competitive performance, though domain-specific models like ESM-2 achieved higher Fmax scores.
  • Binding Affinity Prediction: Outperformed specialized baselines on the BindingDB dataset, demonstrating the advantage of the unified representation.
  • Covalent Binder Prediction: Achieved 70.8% accuracy for predicting covalent adduct SMILES and 89.2% accuracy for identifying the adduct position within the protein sequence. This is a significant result on a novel benchmark.

Limitations and Future Work

  • Protein Function Prediction Performance: While competitive, T5ProtChem did not surpass specialized models like ESM-2 in protein function prediction, suggesting room for improvement in this specific area.
  • Covalent Binder Limitations: The current covalent binder task is limited to singly-bonded adducts and excludes adducts bonding to two amino acids, due to limitations in the GroupSELFIES representation.
  • Future Directions: The authors suggest that this unified framework can be extended to other interdisciplinary tasks in drug discovery, protein engineering, and computational biology. The model's ability to handle diverse inputs and outputs makes it a versatile tool for future research.

Summary updated

BibTeX
@article{kelly2025unified,
  author = {Thomas Kelly and Song Xia and Jieyu Lu and Yingkai Zhang},
  title = {{Unified Deep Learning of Molecular and Protein Language Representations with T5ProtChem}},
  journal = {Journal of Chemical Information and Modeling},
  year = {2025},
  doi = {10.1021/acs.jcim.5c00051},
}
Figure