Abstract
Recently, numerous efforts have continued to push up performance boundaries of document-level relation extraction (DocRE) and have claimed significant progress in DocRE. In this paper, we do not aim at proposing a novel model for DocRE. Instead, we take a closer look at the field to see if these performance gains are actually true. By taking a comprehensive literature review and a thorough examination of popular DocRE datasets, we find that these performance gains are achieved upon a strong or even untenable assumption in common: all named entities are perfectly localized, normalized, and typed in advance. Next, we construct four types of entity mention attacks to examine the robustness of typical DocRE models by behavioral probing. We also have a close check on model usability in a more realistic setting. Our findings reveal that most of current DocRE models are vulnerable to entity mention attacks and difficult to be deployed in real-world end-user NLP applications. Our study calls more attentions for future research to stop simplifying problem setups, and to model DocRE in the wild rather than in an unrealistic Utopian world.
Introduction
Document-level Relation Extraction (DocRE) is a critical task in natural language processing that involves identifying semantic relationships between entities across multiple sentences within a single document. As the field has moved beyond simple sentence-level extraction, recent benchmarks have reported impressive performance gains. However, a significant disconnect exists between these laboratory successes and real-world utility.
Most existing DocRE research operates under a "Utopian" assumption: that at test time, every entity mention has been perfectly localized, normalized (linked to a unique identifier), and typed by a human or a flawless preprocessing system. In practice, any automated pipeline must first employ Named Entity Recognition (NER) and Entity Linking (EL) tools, both of which are inherently error-prone. This paper investigates whether current DocRE models are robust enough to survive the "messy" data encountered in real-world applications.
Methodology
To evaluate the state of the field, we conducted a three-part reality check:
1. Literature and Dataset Analysis
We reviewed 27 recent DocRE models and three major datasets: DocRED (Wikipedia-based), CDR (Chemical-Disease Relations), and GDA (Gene-Disease Associations). This review confirmed that the vast majority of models treat DocRE as a pairwise classification task rather than a true extraction task, relying on pre-provided "gold" entity information.
2. Behavioral Probing via Entity Mention Attacks
To simulate the errors produced by imperfect upstream NER and EL systems, we designed four types of "attacks" to probe model robustness:
- Drop Attack (DrpAtt): Randomly removes 50% of an entity's mentions to simulate missed detections.
- Boundary Attack (BryAtt): Slightly shifts the start or end positions of mentions to simulate imprecise localization.
- Coreference Attack (CorAtt): Intentionally mislinks mentions to the wrong entity to simulate linking errors.
- Mixed Attack (MixAtt): A combination of the three strategies above.
3. Usability Evaluation
We tested the compatibility of popular, off-the-shelf NER and EL tools (including Flair, spaCy, and Stanza) with current DocRE models to determine if these tools can actually generate the high-quality input format these models require.
Key Insights
The Fragility of Modern Architectures
Our analysis reveals that high-performing BERT-based architectures are paradoxically the most vulnerable to mention attacks. These models rely heavily on precise entity markers (e.g., [entity] tags) inserted into the text. When these markers are slightly displaced or missing, the model's internal representation of the document's structure collapses, leading to a significant drop in performance.
The Complexity of Entity Aliasing
Document-level extraction is significantly more complex than sentence-level tasks due to the sheer variety of entity mentions. In the GDA dataset, a single disease entity can have up to 778 unique aliases. Current models assume these aliases are perfectly grouped beforehand, ignoring the primary challenge of document-level understanding: resolving coreference and linking across long-form text.
The Pipeline Bottleneck
There is a massive gap between the output of standard NER/EL tools and the input requirements of DocRE models. We found that the "reproduction rate"—the ability of automated tools to provide the specific entity formats required by DocRE models—ranges only between 34.3% and 58.1%. This suggests that current models are essentially unusable in a standard automated pipeline.
Results
Performance Collapse
Under the Mixed Attack (MixAtt), F1 scores for top-tier models plummeted by amounts ranging from 7.93% to 85.51%, depending on the dataset. The Coreference Attack (CorAtt) proved to be the most damaging, indicating that DocRE models are extremely sensitive to how mentions are grouped into entities.
Tooling Inadequacy
Off-the-shelf NER systems achieved only approximately 63% F1 on the DocRED dataset. Entity linking performance was even lower; the best-performing system (TagMe) reached only a 38.7% F1 score. These results highlight that the "perfect input" required by modern DocRE models is currently unavailable through existing technology.
Conclusion and Future Work
This study serves as a call to action for the NLP community to move beyond "Utopian" benchmarks. To bridge the gap between research and reality, we propose two primary shifts in focus:
- Realistic Benchmarking: Future benchmarks should stop providing "gold" entities at test time and instead evaluate models on raw or noisily-tagged text.
- End-to-End Modeling: Research should prioritize the development of models that jointly extract entities and relations ("DocRE in the wild"), rather than relying on a fragile pipeline of isolated modules.
By addressing these challenges, the field can move toward document-level extraction systems that are truly deployable in real-world end-user applications.
Summary updated