Multilevel Attention Network with Semi-supervised Domain Adaptation for Drug-Target Prediction

Zhousan Xie, Shikui Tu, Lei Xu

Abstract

Predicting drug-target interactions (DTI) is a cornerstone of modern drug discovery. However, computational models often struggle with data scarcity, shortcut learning biases, and domain shifts. This paper presents MlanDTI, a novel framework that integrates pre-trained language models with a multilevel attention mechanism and a robust semi-supervised domain adaptation strategy. By fusing features across all encoder layers, MlanDTI captures hierarchical protein structures more effectively, mitigating biases and significantly improving performance in cross-domain scenarios.

Introduction

The identification of interactions between drug molecules and target proteins is essential for virtual screening and drug repurposing. Despite the potential of deep learning, current methods face significant challenges. Proteins are complex, high-dimensional entities, while drugs are relatively simple. This disparity often leads models to rely on "shortcut learning"—memorizing drug patterns while ignoring the protein's structural nuances. Furthermore, models often fail when applied to new chemical domains (domain shift) due to the limited availability of labeled DTI data.

The MlanDTI Framework

MlanDTI addresses these challenges through a multi-faceted architectural approach that combines large-scale pre-training with hierarchical feature alignment.

The MlanDTI architecture overview, including (a) the main pipeline with BERT embeddings and multilevel attention, (b) the feature fusing mechanism, and (c) the semi-supervised domain adaptation loop.

1. Pre-trained BERT Embeddings

The model leverages large-scale pre-trained models—ChemBERTa-2 for drug SMILES sequences and ProtTrans for protein amino acid sequences. These models provide a rich, context-aware representation of chemical and biological "grammar" learned from millions of unlabeled sequences, providing a strong foundation for downstream prediction tasks.

2. Multilevel Cross-Attention (Mlan)

Unlike traditional models that only utilize the final layer of an encoder, MlanDTI implements a multilevel attention mechanism. It fuses features from every layer of the protein encoder, allowing the model to align drug features with protein structures at multiple scales, from local amino acid motifs to global 3D folds. This hierarchical alignment is key to capturing the true nature of drug-protein interactions.

3. Semi-supervised Domain Adaptation

To improve generalization to unseen domains, MlanDTI employs a pseudo-labeling strategy. The model predicts interactions on unlabeled target-domain data and uses high-confidence predictions as training labels. To mitigate the risk of "confirmation bias" from noisy labels, an auxiliary classifier provides a "second opinion" based on raw BERT outputs, making the pseudo-labeling process much more robust.

Detail of the semi-supervised domain adaptation process, showing the primary and auxiliary classifiers used for robust pseudo-label selection.

Experimental Evaluation

MlanDTI was evaluated on several benchmarks, including Human, C. elegans, BindingDB, and BioSNAP. To simulate real-world discovery, the authors used clustering-based splits to ensure that test sets contained drugs and proteins chemically distinct from the training data.

Key Results

  • State-of-the-Art Performance: MlanDTI consistently outperformed established baselines like MolTrans and DrugBAN across all datasets.
  • Robustness to Domain Shift: In challenging cross-domain tests, MlanDTI achieved improvements of 11.52% in AUROC on BindingDB and 11.29% on BioSNAP compared to previous state-of-the-art models.
  • Ablation Studies: Experiments confirmed that both the BERT embeddings and the multilevel attention mechanism are critical; removing either component led to a significant drop in predictive accuracy.

Key Insights

  • Hierarchical Alignment: Proteins possess hierarchical structures (primary sequence to tertiary folds). By integrating features from all encoder layers, MlanDTI ensures that the model can find the optimal level of interaction for a given drug-protein pair.
  • Mitigating Shortcut Learning: Forcing the model to incorporate multilevel protein features prevents it from "cheating" by only looking at drug patterns, leading to better generalization on unseen proteins.
  • The Power of Auxiliary Classifiers: In domain adaptation, models often get stuck in a loop with wrong pseudo-labels. By using an auxiliary classifier that looks directly at the raw BERT outputs, MlanDTI creates a robust verification step that reduces labeling noise.

Conclusion

MlanDTI represents a significant step forward in DTI prediction. By combining the power of pre-trained biological language models with hierarchical feature fusion and robust domain adaptation, it overcomes several fundamental limitations of previous AI-driven drug discovery tools. Future work may involve incorporating explicit 3D structural data and further refining the semi-supervised learning process to handle even more diverse chemical domains.

Summary updated

Abstract

Prediction of drug-target interactions (DTIs) is a crucial step in drug discovery, and deep learning methods have shown great promise on various DTI datasets. However, existing approaches still face several challenges, including limited labeled data, hidden bias issue, and a lack of generalization ability to out-of-domain data. These challenges hinder the model's capacity to learn truly informative interaction features, leading to shortcut learning and inferior predictive performance on novel drug-target pairs. To address these issues, we propose MlanDTI, a semi-supervised domain adaptive multilevel attention network (Mlan) for DTI prediction. We utilize two pre-trained BERT models to acquire bidirectional representations enriched with information from unlabeled data. Then, we introduce a multilevel attention mechanism, enabling the model to learn domain-invariant DTIs at different hierarchical levels. Moreover, we present a simple yet effective semi-supervised pseudo-labeling method to further enhance our model's predictive ability in cross-domain scenarios. Experiments on four datasets show that MlanDTI achieves state-of-the-art performances over other methods under intra-domain settings and outperforms all other approaches under cross-domain settings. The source code is available at https://github.com/CMACH508/MlanDTI.
BibTeX
@inproceedings{xie2024multilevel,
  author = {Zhousan Xie and Shikui Tu and Lei Xu},
  title = {{Multilevel Attention Network with Semi-supervised Domain Adaptation for Drug-Target Prediction}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i1.27786},
}
Figure