Extracting Drug-Drug and Protein-Protein Interactions from Text using a Continuous Update of Tree-Transformers

Sudipta Roy, Robert E. Mercer2023 · DOI

Abstract

Understanding biological mechanisms requires determining mutual protein-protein interactions (PPI). Obtaining drug-drug interactions (DDI) from scientific articles provides important information about drugs. Extracting such medical entity interactions from biomedical articles is challenging due to complex sentence structures. To address this issue, our proposed model utilizes tree-transformers to generate the sentence representation first, and then a sentence-to-word update step to fine-tune the word embeddings which are again used by the tree-transformers to generate enriched sentence representations. Using the tree-transformers helps the model preserve syntactical information and provide semantic information. The fine-tuning provided by the continuous update step adds improved semantics to the representation of each sentence. Our model outperforms other prominent models with a significant performance boost on the five standard PPI corpora and a performance boost on the one benchmark DDI corpus that are used in our experiments.

Introduction

The explosion of biomedical literature makes it impossible for humans to manually track every discovered interaction between proteins or drugs. Automated Relation Extraction (RE) is the solution, but biomedical text is notoriously difficult. Sentences in these papers are often long, syntactically complex, and feature "long-distance dependencies"—where two interacting entities (like two proteins) are separated by dozens of words. Standard models that read text as a simple sequence often lose the structural "map" of the sentence, leading to missed or incorrect interactions.

To evaluate the proposed solution, the model was tested on two distinct tasks: PPI Extraction using five standard corpora (AIMed, BioInfer, IEPA, HPRD50, and LLL) and DDI Extraction using the SemEval-2013 DDIExtraction dataset. To ensure the model focused on relationships rather than specific names, "entity blinding" was employed, replacing names with generic tags like PROTEIN1 or DRUG0.

Methods

The proposed model combines structural syntax with the power of Transformers through several key components:

  1. Dual Tree-Transformers: Instead of linear processing, the model uses Constituency Trees to capture phrase-level hierarchy and Dependency Trees to capture grammatical relations regardless of distance.
  2. Bio-RoBERTa Embeddings: The system initializes with high-quality, pre-trained biomedical word representations.
  3. The Continuous Update Module: This innovation involves a "re-reading" loop. The model generates an initial sentence representation, then uses a Heterogeneous Graph Attention Network (H-GAT) to pass that sentence-level information back down to individual word embeddings. A second pass through the Tree-Transformers is then performed using these context-enriched meanings.
  4. Multi-branch Attention: Six attention heads and Position-wise Convolutional Neural Networks (PCNNs) are used to capture diverse semantic sub-spaces.
Architecture of the proposed model featuring Tree-Transformers and the Sentence-to-Word Update Module.

The "sentence-to-word" update step acts like a human re-reading a difficult sentence. Once the "gist" is understood, specific words are re-examined with a better understanding of their role. This "fine-tuning on the fly" adds improved semantics without needing to fine-tune millions of parameters in the base BERT model. By using both constituency and dependency trees, the model preserves the "shape" of the sentence, which is crucial for determining if an interaction is positive, negative, or hypothetical.

Results

The proposed model outperformed previous state-of-the-art (SOTA) models—including sequential, tree-structured, and graph-based architectures—by significant margins.

Performance comparison (F1-score) of the proposed model against previous state-of-the-art architectures on five PPI datasets.

On the PPI task, the model achieved an average F1-score of 94.82% across five datasets. Notably, on the difficult AIMed corpus, it reached 94.66%, a 6.39 percentage point boost over the previous SOTA. For the DDI task, it achieved a 95.2% F1-score, surpassing the previous best by 1.8 percentage points. The model showed high precision and recall across all interaction types (Mechanism, Effect, Advice, and Interaction). Ablation studies confirmed that removing the update module caused a significant performance drop (5.12 percentage points across PPI corpora), validating the "re-reading" loop as a primary driver of success.

Conclusion

While the model requires more computational resources due to its dual-pass architecture, it demonstrates superior performance in capturing complex biomedical relations. Future work may extend this approach to document-level relations to capture interactions described across multiple sentences and incorporate external knowledge graphs to further enhance performance.

Summary updated

Abstract

Understanding biological mechanisms requires determining mutual protein-protein interactions (PPI). Obtaining drug-drug interactions (DDI) from scientific articles provides important information about drugs. Extracting such medical entity interactions from biomedical articles is challenging due to complex sentence structures. To address this issue, our proposed model utilizes tree-transformers to generate the sentence representation first, and then a sentence-to-word update step to fine-tune the word embeddings which are again used by the tree-transformers to generate enriched sentence representations. Using the tree-transformers helps the model preserve syntactical information and provide semantic information. The fine-tuning provided by the continuous update step adds improved semantics to the representation of each sentence. Our model outperforms other prominent models with a significant performance boost on the five standard PPI corpora and a performance boost on the one benchmark DDI corpus that are used in our experiments.
BibTeX
@misc{roy2023extracting,
  author = {Sudipta Roy and Robert E. Mercer},
  title = {{Extracting Drug-Drug and Protein-Protein Interactions from Text using a Continuous Update of Tree-Transformers}},
  year = {2023},
  doi = {10.18653/v1/2023.bionlp-1.25},
}
Figure