Abstract
Document-level relation extraction (RE) aims at extracting relations among entities expressed across multiple sentences, which can be viewed as a multi-label classification problem. In a typical document, most entity pairs do not express any pre-defined relation and are labeled as "none" or "no relation". For good document-level RE performance, it is crucial to distinguish such none class instances (entity pairs) from those of pre-defined classes (relations). However, most existing methods only estimate the probability of pre-defined relations independently without considering the probability of "no relation". This ignores the context of entity pairs and the label correlations between the none class and pre-defined classes, leading to sub-optimal predictions. To address this problem, we propose a new multi-label loss that encourages large margins of label confidence scores between each pre-defined class and the none class, which enables captured label correlations and context-dependent thresholding for label prediction. To gain further robustness against positive-negative imbalance and mislabeled data that could appear in real-world RE datasets, we propose a margin regularization and a margin shifting technique. Experimental results demonstrate that our method significantly outperforms existing multi-label losses for document-level RE and works well in other multi-label tasks such as emotion classification when none class instances are available for training.
Introduction
Relation Extraction (RE) involves identifying semantic relationships between entities in text. Document-level RE is particularly challenging because relations often span multiple sentences. A significant hurdle in this task is the extreme data imbalance: the vast majority of entity pairs in a document do not share a predefined relationship and are categorized into the "none" class.
Standard approaches typically employ Binary Cross Entropy (BCE) loss, treating each relation as an independent binary classification. This approach suffers from two primary drawbacks. First, it relies on a global, fixed threshold (e.g., score > 0.5) to determine the presence of a relation, failing to account for the specific context of different entity pairs. Second, it ignores the inherent correlations between the "none" class and predefined relations; an increase in the probability of "none" should logically decrease the probability of other relations.
Methodology: None Class Ranking Loss (NCRL)
To overcome these limitations, we propose the None Class Ranking Loss (NCRL). This method reframes relation extraction as a ranking task where the "none" class serves as a dynamic, instance-dependent anchor.
Adaptive Thresholding and Margin Maximization
Instead of a static global threshold, NCRL learns a score for the "none" class, denoted as $f_0$. During inference, any relation $i$ with a score $f_i > f_0$ is predicted as positive. The loss function is designed to maximize the margin between the "none" class score and other classes. Specifically, it pushes scores of positive labels significantly above $f_0$ and scores of negative labels below it.
Robustness Enhancements
To handle the challenges of real-world datasets, we introduce two additional techniques:
- Margin Regularization: This prevents the model from excessively suppressing negative samples, which is vital when positive samples are extremely sparse.
- Margin Shifting: This introduces a buffer to improve robustness against mislabeled data, a common issue in distantly supervised datasets where labels are automatically generated.
Experimental Evaluation
We evaluated NCRL on three major benchmarks: DocRED (Wikipedia-based document RE), DialogRE (dialogue-based RE), and GoEmotions (fine-grained emotion classification).
Key Results on DocRED
On the DocRED leaderboard, NCRL achieved state-of-the-art results. When combined with DeBERTa-Large, it reached an $Ign F_1$ score of 65.81% on the test set, significantly outperforming standard BCE and the previous state-of-the-art Adaptive Thresholding Loss (ATL).
Generalization and Robustness
Beyond RE, NCRL improved mean Average Precision (mAP) on the GoEmotions dataset by 1.49% over BCE, demonstrating its effectiveness as a general-purpose multi-label loss. Furthermore, NCRL showed superior stability when trained on noisy, distantly supervised data compared to existing methods, validating the efficacy of the margin shifting technique.
Conclusion and Future Work
NCRL effectively addresses the "none" class imbalance in document-level RE by treating it as a context-dependent threshold. While the method is architecture-agnostic and yields significant performance gains, future work could explore its interaction with specific neural architectures like Graph Neural Networks. Additionally, extending the framework to handle hierarchical label structures or "Open RE" scenarios remains an open area for research.
Summary updated