TL;DR
Document-level relation extraction models often struggle to predict rare relations or handle entity pairs that express multiple relationships at once. This paper introduces Relation Correlations—the concept that real-world relationships naturally co-occur—using two novel co-occurrence prediction sub-tasks to share knowledge between data-rich and data-scarce relations. This approach significantly boosts performance, yielding up to a 12.39% F1 improvement on complex multi-label extractions.
What Problem Does This Paper Address?
Document-level Relation Extraction (DocRE) requires a model to identify relationships between all entity pairs across an entire document. Unlike sentence-level extraction, DocRE is highly realistic—over 40% of relational facts in standard datasets can only be extracted by connecting clues across multiple sentences.
However, current DocRE models hit a wall when facing two major, overlooked challenges:
- The Long-Tail Problem: A few relations have thousands of training examples, but the vast majority have very few (e.g., less than 200). Standard training causes models to severely underfit these rare relations.
- The Multi-Label Problem: Around 7% of entity pairs express multiple relations simultaneously (e.g., a person's country of citizenship and their place of birth). Classifying these requires drawing highly complex, overlapping decision boundaries.
Existing works focus almost entirely on building better contextual representations of text, treating relation labels as completely independent. This paper argues that relations are inherently interdependent, and modeling their correlations can solve both bottlenecks.
Methods
The authors propose a model-agnostic framework that learns Relation Correlations through two auxiliary sub-tasks, using relation embeddings as a bridge:
- Joint Embedding Learning (JE): Instead of just passing the document text to a Pre-trained Language Model (like BERT), the authors append a sequence of all possible relation labels to the text input:
[CLS] document tokens [SEP] relation_1, relation_2... [SEP]. This forces the model to map text and labels into the exact same hidden space, allowing self-attention layers to automatically capture dependencies between them. - Coarse-grained Relation Co-occurrence Prediction (CRCP): This label-level sub-task trains the model to predict whether a specific target relation is likely to appear given a set of other expressed relations in the document, independent of the specific entities.
- Fine-grained Relation Co-occurrence Prediction (FRCP): Because relationships ultimately depend on specific entity pairs, this sub-task introduces a Relation-Aggregated Embedding (RAE). It dynamically weights and aggregates relation embeddings based on token-to-relation attention scores to predict co-occurrences conditioned on both the text context and the entity pair.
During final inference, these correlation-aware relation embeddings are fused back into the entity pair features to guide the final relation classifier.
Datasets
The model was evaluated on two widely recognized document-level relation extraction benchmarks:
- DocRED: A large-scale, crowdsourced dataset built from Wikipedia articles containing 96 relation types.
- DWIE: An entity-centric dataset for multi-task document-level information extraction containing 65 relation types.
Key Insights and Ideas
- Knowledge Transfer via Anchors: Data-rich "head" relations can act as anchors to support data-scarce "tail" relations. If the model learns that date of birth strongly correlates with place of birth, it can leverage the abundance of the former to improve predictions for the latter when training data is scarce.
- Semantic Distance as a Delineator: For multi-label pairs, co-occurrence frequency acts as a proxy for semantic distance. Knowing which relations frequently overlap helps the classifier narrow down its search space and draw sharper decision boundaries.
Key Results
The proposed model, BERT-Correl<sub>BASE</sub>, consistently outperformed strong baselines (including ATLOP and GAIN) across both datasets:
- Mitigating the Long-Tail: On data-scarce relations with fewer than 100 training instances, the model improved the macro-F1 score by up to 6.49% on DocRED and 6.12% on DWIE.
- Conquering Multi-Labels: For entity pairs expressing four distinct relations simultaneously, the model achieved a massive 12.39% F1 improvement over the baseline.
- Captured vs. Statistical Correlations: Visualizations of the learned embedding similarities closely mirrored the actual statistical co-occurrence matrices of the datasets, proving that the auxiliary tasks successfully captured real-world semantic links.
Limitations and Future Work
The authors do not explicitly detail limitations in the text, but a few architectural constraints are apparent:
- Sequence Length Overhead: Appending all relation labels to the input text sequence increases the token count. For datasets with hundreds of relation types, this could easily exceed the standard 512-token limit of models like BERT.
- Noise in Learned Matrices: While the learned correlation matrices closely match statistical realities, they still contain minor noise points that could occasionally misguide the classifier. Future work could explore filtering or regularizing these correlation matrices.
Summary updated