Abstract
Recently, numerous efforts have continued to push up performance boundaries of document-level relation extraction (DocRE) and have claimed significant progress in DocRE. In this paper, we do not aim at proposing a novel model for DocRE. Instead, we take a closer look at the field to see if these performance gains are actually true. By taking a comprehensive literature review and a thorough examination of popular DocRE datasets, we find that these performance gains are achieved upon a strong or even untenable assumption in common: all named entities are perfectly localized, normalized, and typed in advance. Next, we construct four types of entity mention attacks to examine the robustness of typical DocRE models by behavioral probing. We also have a close check on model usability in a more realistic setting. Our findings reveal that most of current DocRE models are vulnerable to entity mention attacks and difficult to be deployed in real-world end-user NLP applications. Our study calls more attentions for future research to stop simplifying problem setups, and to model DocRE in the wild rather than in an unrealistic Utopian world.
Introduction
Document-Level Relation Extraction (DocRE) is a critical task in natural language processing, requiring the identification of semantic relationships between entities that often span multiple sentences. Unlike sentence-level extraction, DocRE necessitates reasoning over long-range dependencies and coreference resolution. While recent leaderboards show impressive performance gains, this paper provides a sobering "reality check," questioning whether these models are truly ready for real-world deployment.
The "Utopian" Assumption
The primary finding of this study is that the research community has been inadvertently simplifying the DocRE problem. Most state-of-the-art (SOTA) models are evaluated under what the authors call a "Utopian" assumption: all entity mentions are perfectly localized, normalized to unique IDs, and typed before the relation extraction model even begins its work.
In practice, this means that the "extraction" task has been reduced to a "pairwise classification" task. The hardest parts of the pipeline—identifying every mention of an entity (e.g., "Juan Guzmán," "Hans Gutmann," "he") and linking them correctly—are bypassed by using gold-standard annotations.
Dataset Analysis and Complexity
The study examines three major benchmarks: DocRED (Wikipedia-based), CDR (Chemical-Disease), and GDA (Gene-Disease). A key characteristic of these datasets is the high number of mentions involved in each relation instance, which significantly increases the complexity of the task.
As shown in the figure above, a significant portion of relations involves more than 10 mentions. In the GDA dataset, some entities have hundreds of unique aliases. Current models assume every single one of these is correctly identified and linked, which is a highly unrealistic expectation for real-world NLP tools.
Methodology: Probing Robustness and Usability
To test the robustness of current models, the authors designed four types of "entity mention attacks" to simulate errors from upstream tools:
- DrpAtt (Drop Attack): Randomly dropping 50% of mentions.
- BryAtt (Boundary Attack): Slightly shifting mention boundaries.
- CorAtt (Coreference Attack): Mis-linking mentions to the wrong entity.
- MixAtt (Mixed Attack): A combination of the above.
Additionally, they conducted a usability check by replacing gold-standard entities with outputs from off-the-shelf Named Entity Recognition (NER) and Entity Linking (EL) systems like Flair, spaCy, and TagMe.
Key Results: A Performance Collapse
The results reveal extreme brittleness in SOTA models. Under the MixAtt scenario, the F1 score of a standard BERT-Marker model plummeted by over 80% on biomedical datasets. The CorAtt (coreference) attack was particularly devastating, highlighting that these models rely almost entirely on perfect entity linking.
Furthermore, the "tooling gap" is substantial. Even top-tier NER systems like Flair only achieve an F1 of ~63% on DocRED, far below the "perfect" input models are trained on. When using these real-world tools, the "reproduction rate" of the required input format for DocRE models is only between 34% and 58%, making benchmark scores currently unachievable in practice.
Conclusion and Future Directions
This study calls for the NLP community to move beyond "Utopian" benchmarks and focus on "DocRE in the wild." The authors suggest three paths forward:
- End-to-End Modeling: Developing models that jointly extract entities and relations to avoid pipeline bottlenecks.
- Robustness Training: Training models to handle noisy or missing mentions.
- Realistic Evaluation: Creating benchmarks that do not provide gold-standard entity information at test time.
Summary updated