TL;DR
Syn2DRE is a novel semi-supervised framework for Document-level Relation Extraction (DocRE) that overcomes the "confirmation bias" of traditional pseudo-labeling. By using a dual teacher-student architecture where two different model types (graph-based and transformer-based) cross-supervise each other, the system generates high-quality labels from unlabeled data in a single, efficient training run.
What Problem Does This Paper Address?
Extracting relationships between entities across an entire document (DocRE) is significantly harder than sentence-level extraction because it requires reasoning across multiple sentences. While deep learning models excel at this, they are "data-hungry," and manually annotating DocRE datasets is a nightmare—a single document in the popular DocRED dataset can require annotating roughly 38,000 candidate entity triples.
To bypass this, researchers use pseudo-labeling: training a model on a small labeled set, using it to label a large unlabeled set, and then retraining. However, this often leads to confirmation bias, where the model reinforces its own mistakes. Existing solutions to this bias are often computationally expensive, requiring multiple iterative rounds of "denoising" and retraining.
Methods
The authors introduce Syn2DRE (Synergistic Dual Teacher-Student), which uses two key strategies to produce reliable labels efficiently:
- Dual Teacher-Student Pairs: The framework maintains two separate student models ($s_1, s_2$) and two corresponding teacher models ($t_1, t_2$). The teachers are not trained via backpropagation; instead, they are updated using an Exponential Moving Average (EMA) of their respective students' weights. This creates a "stable" version of the model that provides more reliable targets.
- Unsupervised Synergistic Cross-supervision: To break the cycle of confirmation bias, the authors use two different architectures: KD-DocRE (transformer-based) and UGDRE (graph-based). During training, these models exchange "disagreeing" pseudo-labels. If Model A is highly confident about a relation where Model B is uncertain or disagrees, Model B learns from Model A’s prediction.
- Uncertainty-Aware Selection: Not all pseudo-labels are kept. The system uses Monte Carlo (MC) dropout to estimate uncertainty. If a teacher's predictions vary wildly across multiple forward passes, that label is discarded as unreliable.
Datasets
The framework was evaluated on five major benchmarks:
- General Domain: DocRED and its refined version, Re-DocRED (English), and HacRED (Chinese).
- Biomedical Domain: CDR (Chemical-Disease Relations) and GDA (Gene-Disease Associations).
Key Insights and Ideas
- Architecture Diversity as a Filter: The most powerful idea here is that different model architectures (graph vs. transformer) have different "blind spots." By forcing them to teach each other specifically on instances where they disagree, the framework effectively filters out errors that a single model type would likely repeat.
- Efficiency via Single-Pass Training: Unlike prior state-of-the-art methods that require multiple rounds of training and denoising, Syn2DRE achieves better results in a single end-to-end run.
- Robustness to the "Long Tail": DocRE datasets are notoriously imbalanced, with a few common relations and many rare ones. Syn2DRE's ability to generate reliable synthetic labels significantly boosts performance on these infrequent "long-tail" relations.
Key Results
Syn2DRE achieved state-of-the-art (SOTA) results across all tested datasets:
- DocRED/Re-DocRED: Consistently outperformed previous SOTA methods like UGDRE by +0.38 to +1.26 F1 points.
- Biomedical Gains: The most dramatic improvement was on the CDR dataset, where Syn2DRE boosted the F1 score by 8.47 points over the previous best baseline.
- Long-tail Performance: On rare relations in Re-DocRED, it achieved an LT $F_1$ of 74.90, significantly higher than the 72.88 of the next best method (DURE).
- Computational Efficiency: Syn2DRE completed training in 26 hours, compared to 55 hours for UGDRE, while using only about 30% of the unlabeled data required by its predecessor.
Limitations and Future Work
Despite its success, the authors acknowledge a few hurdles:
- Memory Overhead: Running two teacher-student pairs simultaneously nearly doubles the GPU memory requirement compared to single-model frameworks.
- Labeled Data Dependency: While it uses unlabeled data effectively, it still requires a "non-trivial" amount of human-annotated data for the initial warm-up phase.
- Fixed Thresholds: The current system uses fixed confidence thresholds for selecting pseudo-labels; the authors suggest that adaptive thresholds (learning when to trust the model) could be a promising future direction.
Summary updated