A Novel Document-Level Relation Extraction Method Based on BERT and Entity Information

Xiaoyu Han, Lei WangIEEE Access 2020 · DOI

Abstract

Document-level relation extraction (RE) aims to identify relationships between entities across multiple sentences in a paragraph. Unlike sentence-level RE, document-level tasks involve longer texts and a higher density of entities, making it significantly more challenging. This paper introduces the Document-level Entity Mask Method with Type information (DEMMT), which utilizes special tokens to mask entity mentions and provide explicit type and identity cues. Built upon DEMMT, a BERT-based one-pass model is proposed to predict all entity relationships in a single processing step. Evaluations on the DocRED dataset demonstrate that this approach achieves substantial F1 score improvements over both non-pretrained state-of-the-art models and standard BERT implementations.

Introduction

Traditional Relation Extraction (RE) models primarily focus on sentence-level tasks. However, in real-world documents, critical information is often distributed across multiple sentences. Document-level RE introduces three primary challenges:

  1. Inter-sentential Relations: In datasets like DocRED, over 40% of relational facts require cross-sentence reasoning.
  2. Entity Complexity: Documents often contain numerous entities, each appearing as multiple mentions with varying surface forms.
  3. Computational Efficiency: Iteratively processing every possible entity pair in a document is computationally prohibitive as the number of entities grows.

Methodology: DEMMT and the One-Pass Architecture

To address these challenges, the authors propose a two-part solution: the DEMMT masking strategy and a one-pass BERT-based model.

DEMMT (Document-level Entity Mask Method with Type information)

Instead of processing raw text, DEMMT wraps every entity mention with special tokens that inject metadata directly into the input sequence:

  • Type Information: A prefix token (e.g., [ORG], [PER]) indicates the entity's semantic category.
  • Identity Information: A suffix token (e.g., [MASK_1]) uniquely identifies which specific entity the mention belongs to.

This allows the model to easily link disparate mentions of the same entity across a long document.

One-Pass Model Architecture

The architecture is designed for efficiency, processing the entire document once to extract all relations.

Architecture of the BERT-based one-pass model with DEMMT

As shown in the figure above, the process follows these steps:

  1. Encoding: The document, enriched with DEMMT masks, is passed through a BERT encoder.
  2. Entity Representation: For each entity, the model averages the embeddings of all its mentions to create a unified entity vector.
  3. Bilinear Prediction: A bilinear layer followed by a softmax function predicts the relationships between all possible entity pairs simultaneously.

Experimental Results

The model was evaluated on DocRED, a large-scale open-domain dataset derived from Wikipedia. The dataset includes over 5,000 manually annotated documents and over 100,000 distantly supervised documents.

The BERT + DEMMT model achieved significant performance gains:

  • Manual Data: A 6% F1 improvement over previous non-pretrained state-of-the-art models and a 2% F1 improvement over standard BERT.
  • Distant Supervision: A 5% F1 improvement over pure BERT, highlighting the robustness of the masking method in the presence of noisy labels.

Discussion and Insights

Ablation studies revealed that entity identity (linking mentions) is more critical for performance than entity type. DEMMT essentially provides a "map" for BERT's self-attention mechanism, helping it focus on relevant tokens across long distances. Interestingly, while the method works well with BERT and CNNs, it showed less benefit for LSTMs, likely because LSTMs prioritize global sequence features over the local information provided by special tokens.

Limitations

Despite its efficiency, the model's reliance on BERT's 512-token limit remains a bottleneck for very long documents. Furthermore, while the model captures some relational logic through attention, explicitly modeling complex reasoning paths (e.g., transitivity) remains an area for future work.

Summary updated

Abstract

Document-level relation extraction aims to extract the relationship among the entities in a paragraph of text. Compared with sentence-level, the text in document-level relation extraction is much longer and contains many more entities. It makes the document-level relation extraction a harder task. The number and complexity of entities make it necessary to provide enough information about the entities for the models in document-level relation extraction. To solve this problem, we put forward a document-level entity mask method with type information (DEMMT), which masks each mention of the entities by special tokens. By using this entity mask method, the model can accurately obtain every mention and type of the entities. Based on DEMMT, we propose a BERT-based one-pass model, through which we can predict the relationships among the entities by processing the text once. We test the proposed model on the DocRED dataset, which is a large scale open-domain document-level relation extraction dataset. The results on the manually annotated part of DocRED show that our approach obtains 6% F1 improvement compared with the state-of-the-art models that do not use pre-trained models and has 2% F1 improvement than BERT which does not use the DEMMT. On the distant supervision generated part of DocRED, the improvement of F1 is 2% compared with no pre-trained models, and 5% compared with pure BERT.
BibTeX
@inproceedings{han2020novel,
  author = {Xiaoyu Han and Lei Wang},
  title = {{A Novel Document-Level Relation Extraction Method Based on BERT and Entity Information}},
  booktitle = {IEEE Access},
  year = {2020},
  doi = {10.1109/access.2020.2996642},
}
Figure