None Class Ranking Loss for Document-Level Relation Extraction

Yang Zhou, Wee Sun Lee

TL;DR

This paper introduces None Class Ranking Loss (NCRL), a novel multi-label loss function for document-level relation extraction that treats the "no relation" (none) class as a dynamic, instance-dependent threshold. By maximizing the margin between the none class and pre-defined relations, NCRL captures complex label correlations and provides robust performance even in the face of massive class imbalance and noisy, mislabeled data.

What Problem Does This Paper Address?

Relation Extraction (RE) is the task of identifying semantic connections between entities in text (e.g., "Steve Jobs" founded "Apple"). In document-level RE, these relations often span multiple sentences, making the task significantly harder than sentence-level extraction.

The authors identify a major flaw in current approaches: the "None" class problem. In a typical document, the vast majority of entity pairs have no relation. Most models use Binary Cross Entropy (BCE) to predict each relation independently and then apply a "global threshold" (e.g., if the score is > 0.5, the relation exists).

However, a global threshold is brittle. It ignores the context of specific entity pairs and fails to account for label correlations—specifically, that a high probability of "no relation" should naturally decrease the probability of all other relations. Furthermore, real-world RE datasets are often "noisy," containing many false negatives where a relation exists but wasn't labeled by annotators.

Methods

The core contribution is the None Class Ranking Loss (NCRL). Instead of treating the "none" class as just another label or ignoring it, NCRL uses it as an adaptive anchor.

  1. Ranking Framework: The model learns a confidence score $f_0$ for the none class and scores $f_i$ for all other relations. The loss encourages positive relations to be ranked above $f_0$ and negative relations to be ranked below it.
  2. Implicit Correlation: By forcing relations to compete against the none class score, the model implicitly captures correlations between labels without the computational explosion of comparing every pair of labels directly.
  3. Margin Regularization: To handle the extreme imbalance (way more negatives than positives), the authors add a term that prevents the model from pushing negative scores too far away from the none class, which helps the model stay sensitive to rare positive relations.
  4. Margin Shifting: To combat mislabeled data (false negatives), they introduce a hyperparameter $\gamma$ that "shifts" the margin. This allows the model to ignore "easy" negative samples and reduce the penalty for "hard" negatives that might actually be mislabeled positives.
  5. Theoretical Backing: The authors provide a proof of Bayes consistency, mathematically demonstrating that minimizing NCRL actually leads to the optimal multi-label classification decision.

Datasets

The authors evaluated their method on three distinct benchmarks:

  • DocRED: A large-scale Wikipedia-based document RE dataset. They tested on both the human-annotated version and the "distantly supervised" version (which is much larger but very noisy).
  • DialogRE: A dataset for RE in dialogue (based on transcripts from the TV show Friends).
  • GoEmotions: A fine-grained emotion classification dataset (Reddit comments). This was used to prove that NCRL is a general-purpose multi-label loss, not just limited to relation extraction.

Key Insights and Ideas

  • The "None" Class is a Boundary, Not a Label: The most powerful insight is treating the none class score as an instance-specific threshold. This allows the model to be "unsure" about a difficult entity pair by raising the none class score, rather than being forced to pick a relation based on a fixed global cutoff.
  • Efficiency through Ranking: Traditional ranking losses often require $O(K^2)$ comparisons (comparing every positive label to every negative label). NCRL achieves similar benefits with $O(K)$ complexity by using the none class as the sole point of comparison.
  • Robustness to Noise: The "Margin Shifting" technique is particularly effective for distantly supervised data. It essentially tells the model: "If you're very confident this is a negative, don't worry about it; but if it's a 'hard' negative, be careful, it might be a missing label."

Key Results

  • State-of-the-Art Performance: On the DocRED leaderboard, NCRL combined with a DeBERTa-large encoder achieved an Ign $F_1$ score of 65.81%, significantly outperforming previous methods like ATL (Adaptive Thresholding Loss) and standard BCE.
  • Consistency Across Tasks: On DialogRE, NCRL improved $F_1$ scores by roughly 2% over BCE. On GoEmotions, it improved mean Average Precision (mAP) by 1.49%, showing its utility in general NLP tasks.
  • Noise Tolerance: When training on the noisy, distantly supervised DocRED data, NCRL showed much higher gains than other losses, proving that the margin shifting technique successfully mitigates labeling errors.

Limitations and Future Work

  • Dependency on "None" Instances: NCRL requires the training set to contain explicit "none" class instances (entity pairs with no relations). In datasets where every instance is guaranteed to have at least one positive label (like the AAPD dataset), the none class score $f_0$ cannot be properly estimated, and the model reverts to needing a tuned threshold.
  • Hyperparameter Tuning: While the model is robust, the margin shifting parameter $\gamma$ and the regularization weight still require tuning for different datasets to achieve optimal results.
  • Future Directions: The authors suggest exploring how NCRL could be adapted for "open-world" settings where new relations might appear that weren't in the training set.

Summary updated

Abstract

Document-level relation extraction (RE) aims at extracting relations among entities expressed across multiple sentences, which can be viewed as a multi-label classification problem. In a typical document, most entity pairs do not express any pre-defined relation and are labeled as "none" or "no relation". For good document-level RE performance, it is crucial to distinguish such none class instances (entity pairs) from those of pre-defined classes (relations). However, most existing methods only estimate the probability of pre-defined relations independently without considering the probability of "no relation". This ignores the context of entity pairs and the label correlations between the none class and pre-defined classes, leading to sub-optimal predictions. To address this problem, we propose a new multi-label loss that encourages large margins of label confidence scores between each pre-defined class and the none class, which enables captured label correlations and context-dependent thresholding for label prediction. To gain further robustness against positive-negative imbalance and mislabeled data that could appear in real-world RE datasets, we propose a margin regularization and a margin shifting technique. Experimental results demonstrate that our method significantly outperforms existing multi-label losses for document-level RE and works well in other multi-label tasks such as emotion classification when none class instances are available for training.
BibTeX
@inproceedings{zhou2022none,
  author = {Yang Zhou and Wee Sun Lee},
  title = {{None Class Ranking Loss for Document-Level Relation Extraction}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=izwLoMK3ix},
}

Loading related content...

Figure