Abstract
Document-level relation extraction (DocRE) aims to identify relationships between entities within a document. Due to the vast number of entity pairs, fully annotating all fact triplets is challenging, resulting in datasets with numerous false negative samples. Recently, self-training-based methods have been introduced to address this issue. However, these methods are purely black-box and sub-symbolic, making them difficult to interpret and prone to overlooking symbolic interdependencies between relations. To remedy this deficiency, our insight is that symbolic knowledge, such as logical rules, can be used as diagnostic tools to identify conflicts between pseudo-labels. By resolving these conflicts through logical diagnoses, we can correct erroneous pseudo-labels, thus enhancing the training of neural models. To achieve this, we propose LogicST, a neural-logic self-training framework that iteratively resolves conflicts and constructs the minimal diagnostic set for updating models. Extensive experiments demonstrate that LogicST significantly improves performance and outperforms previous state-of-the-art methods. For instance, LogicST achieves an increase of 7.94% in F1 score compared to CAST (Tan et al., 2023a) on the DocRED benchmark (Yao et al., 2019). Additionally, LogicST is more time-efficient than its self-training counterparts, requiring only 10% of the training time of CAST.
Introduction
In Document-Level Relation Extraction (DocRE), an AI must identify relationships between all entities mentioned across an entire document. This is significantly harder than sentence-level extraction because the number of potential entity pairs grows quadratically with the number of entities. Because there are so many pairs, human annotators often miss many valid relationships, leading to "incomplete annotations." When a model is trained on these datasets, it mistakenly learns that these unlabeled-but-true relationships are "negatives" (false negatives).
Current solutions use self-training, where a model predicts "pseudo-labels" for unlabeled pairs and then trains on its own predictions. However, these models are "black boxes" that suffer from confirmation bias: if the model makes a mistake, it reinforces that mistake in the next round of training. They also struggle with long-range dependencies where logic is more important than simple pattern matching.
Methodology: Neural-Symbolic Self-Training
The authors propose LogicST, a framework that integrates symbolic logic into the neural self-training process to identify and correct these errors.
- Teacher-Student Architecture: A "Teacher" model generates pseudo-labels for unlabeled entity pairs. A "Student" model learns from these labels. The Teacher’s weights are then updated using an Exponential Moving Average (EMA) of the Student’s weights to keep training stable.
- Logical Diagnosis: Instead of just accepting the Teacher's pseudo-labels, LogicST uses a set of first-order logical rules (implication, composition, and negation) to find conflicts. For example, if the model predicts "London is the capital of the UK" but fails to predict "London is located in the UK," the logic engine flags this as a conflict.
- Minimal Diagnosis & Scoring: To resolve conflicts, the system identifies the "minimal" set of label flips needed to satisfy the rules. It uses a novel scoring function that combines model probability with a "Recall Reward"—a dynamic bonus for flipping "False" to "True" early in training, which helps uncover hidden relationships.
- Sequential Diagnosis: To keep the process efficient, LogicST uses a sequential approach that only keeps the top-scoring diagnostic candidates, avoiding the exponential complexity of checking every possible rule combination.
Experimental Results
The framework was evaluated on several major DocRE benchmarks, including DocRED, Re-DocRED, and DWIE. These datasets are known for having significant missing annotations, providing a rigorous test for LogicST's diagnostic capabilities.
Performance and Efficiency
LogicST achieved a 69.26% F1 score on Re-DocRED (using BERT-base), a massive 7.94% absolute improvement over the previous best method (CAST). Beyond accuracy, LogicST is incredibly fast, requiring only 10% of the training time of CAST (approx. 1 hour 52 mins vs. 21 hours 33 mins).
Long-Range Mastery
The model showed its greatest strengths on entity pairs that are far apart in a document. For pairs with a distance of 300+ tokens, LogicST improved F1 scores by 23.67% compared to baselines.
As shown in the figure above, symbolic logic is far better at "connecting the dots" than pure neural patterns when entities are separated by hundreds of tokens. This proves that logic serves as a critical bridge for long-range dependencies.
Conclusion and Future Work
LogicST demonstrates that logic is most effective when used as a diagnostic tool to fix training data, rather than just a constraint during inference. By using a "Recall Reward" strategy, the model effectively "discovers" missing annotations that human labelers overlooked.
While highly effective, the system currently relies on pre-defined logical rules and assumes correct initial Named Entity Recognition (NER). Future work will explore applying this logic-driven self-training to "distant supervision" settings, where labels aren't just missing, but potentially noisy or wrong.
Summary updated