DuRE: Dual Contrastive Self Training for Semi-Supervised Relation Extraction

Yuxi Feng, Laks V. S. LakshmananNAACL 2024 · DOI

Document-level Relation Extraction (RE) involves identifying semantic relationships between entities across multiple sentences in a document. While crucial for knowledge graph construction, this task faces significant hurdles, particularly the "long-tail" problem where rare relations are often neglected by models due to a lack of labeled data. Presented at NAACL 2024, DuRE (Dual Contrastive Self-Training) introduces a novel semi-supervised framework that addresses these challenges by integrating classification and text generation into a unified dual process.

The Long-Tail Problem in Relation Extraction

A primary challenge in RE is the inherent imbalance of real-world datasets. A small number of relations, such as "located in," appear frequently, while many others, like "ethnic group," are extremely rare. Standard models tend to perform well on frequent relations but struggle significantly with the "long tail" of rare ones. Furthermore, manually labeling document-level data is prohibitively expensive, and traditional semi-supervised methods often suffer from confirmation bias, where models reinforce their own errors.

Performance disparity across relations of varying frequencies, highlighting the long-tail problem.

The DuRE Framework

DuRE overcomes these limitations through a Dual Process architecture that jointly trains a document-level RE classifier and a controllable text generator (based on the Flan-T5 architecture).

Dual Learning and Contrastive Loss

Unlike standard self-training, which only generates pseudo-labels for unlabeled text, DuRE uses its generator to create synthetic training data based on known labels. To ensure the quality of this synthetic data, the framework employs a Contrastive Loss (CL). The classifier acts as a "teacher," evaluating the generator's output. If the classifier cannot easily identify the intended relation in the generated text, the generator is penalized. This feedback loop forces the generator to produce more faithful and discriminative training examples.

Self-Adaptive Generation (SAG)

To specifically target the long-tail issue, DuRE incorporates Self-Adaptive Generation (SAG). The model monitors its performance on a development set and adjusts its generation strategy accordingly:

  • Low Recall: The model generates additional synthetic examples for that specific relation to improve coverage.
  • High Precision: The model increases generation "temperature" to introduce diversity and improve generalization.
  • Low Precision: The model decreases temperature to ensure more conservative and accurate text generation.

Key Results and Performance

The authors evaluated DuRE on two benchmarks: Re-DocRED (general domain) and ChemDisGene (CDG) (biomedical domain). DuRE achieved state-of-the-art results in semi-supervised document RE. On Re-DocRED, it reached an F1 score of 76.84, a 2.17-point improvement over the previous best baseline.

The framework's efficiency in low-data regimes is particularly noteworthy. On the specialized CDG dataset, DuRE significantly outperformed baselines across all training set sizes, nearly doubling the performance of the ATLOP-Dual baseline when using only 5% of labeled data.

F1 performance on the CDG dataset across different training set sizes, showcasing low-data efficiency.

In addition to overall performance, DuRE showed substantial gains in "Long-Tail F1" (LT_F1), scoring 72.88 compared to the baseline's 69.34 on Re-DocRED. This confirms that the dual-training approach effectively mitigates the performance drop on rare relations by providing high-quality synthetic examples where human labels are scarcest.

Conclusion and Future Directions

DuRE demonstrates that a closed-loop system, where a classifier and generator mutually improve each other, is highly effective for semi-supervised RE. By surgically augmenting data for rare relations, it provides a robust solution to the long-tail problem. While the computational cost of auto-regressive generation is a consideration, the performance gains—especially in low-resource and specialized domains—make DuRE a significant advancement in the field.

Summary updated

BibTeX
@inproceedings{feng2024dure,
  author = {Yuxi Feng and Laks V. S. Lakshmanan},
  title = {{DuRE: Dual Contrastive Self Training for Semi-Supervised Relation Extraction}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.30},
}
Figure