Abstract
Document-level relation extraction (RE) aims to identify relationships between entities across multiple sentences in a paragraph. Unlike sentence-level RE, document-level tasks involve longer texts and a higher density of entities, making it significantly more challenging. This paper introduces the Document-level Entity Mask Method with Type information (DEMMT), which utilizes special tokens to mask entity mentions and provide explicit type and identity cues. Built upon DEMMT, a BERT-based one-pass model is proposed to predict all entity relationships in a single processing step. Evaluations on the DocRED dataset demonstrate that this approach achieves substantial F1 score improvements over both non-pretrained state-of-the-art models and standard BERT implementations.
Introduction
Traditional Relation Extraction (RE) models primarily focus on sentence-level tasks. However, in real-world documents, critical information is often distributed across multiple sentences. Document-level RE introduces three primary challenges:
- Inter-sentential Relations: In datasets like DocRED, over 40% of relational facts require cross-sentence reasoning.
- Entity Complexity: Documents often contain numerous entities, each appearing as multiple mentions with varying surface forms.
- Computational Efficiency: Iteratively processing every possible entity pair in a document is computationally prohibitive as the number of entities grows.
Methodology: DEMMT and the One-Pass Architecture
To address these challenges, the authors propose a two-part solution: the DEMMT masking strategy and a one-pass BERT-based model.
DEMMT (Document-level Entity Mask Method with Type information)
Instead of processing raw text, DEMMT wraps every entity mention with special tokens that inject metadata directly into the input sequence:
- Type Information: A prefix token (e.g.,
[ORG],[PER]) indicates the entity's semantic category. - Identity Information: A suffix token (e.g.,
[MASK_1]) uniquely identifies which specific entity the mention belongs to.
This allows the model to easily link disparate mentions of the same entity across a long document.
One-Pass Model Architecture
The architecture is designed for efficiency, processing the entire document once to extract all relations.
As shown in the figure above, the process follows these steps:
- Encoding: The document, enriched with DEMMT masks, is passed through a BERT encoder.
- Entity Representation: For each entity, the model averages the embeddings of all its mentions to create a unified entity vector.
- Bilinear Prediction: A bilinear layer followed by a softmax function predicts the relationships between all possible entity pairs simultaneously.
Experimental Results
The model was evaluated on DocRED, a large-scale open-domain dataset derived from Wikipedia. The dataset includes over 5,000 manually annotated documents and over 100,000 distantly supervised documents.
The BERT + DEMMT model achieved significant performance gains:
- Manual Data: A 6% F1 improvement over previous non-pretrained state-of-the-art models and a 2% F1 improvement over standard BERT.
- Distant Supervision: A 5% F1 improvement over pure BERT, highlighting the robustness of the masking method in the presence of noisy labels.
Discussion and Insights
Ablation studies revealed that entity identity (linking mentions) is more critical for performance than entity type. DEMMT essentially provides a "map" for BERT's self-attention mechanism, helping it focus on relevant tokens across long distances. Interestingly, while the method works well with BERT and CNNs, it showed less benefit for LSTMs, likely because LSTMs prioritize global sequence features over the local information provided by special tokens.
Limitations
Despite its efficiency, the model's reliance on BERT's 512-token limit remains a bottleneck for very long documents. Furthermore, while the model captures some relational logic through attention, explicitly modeling complex reasoning paths (e.g., transitivity) remains an area for future work.
Summary updated