Abstract
Relation Extraction (RE) is essential for structured knowledge discovery but typically requires large-scale, manually annotated datasets. In specialized domains, such as medicine or law, the cost of expert annotation is often prohibitive. This paper explores the use of Distant Supervision (DS) to automate labeling, combined with Multi-instance Learning (MIL) to mitigate noise. The authors introduce a novel active learning workflow where domain experts audit the "concepts" (representative trigrams) learned by a Ranking Convolutional Neural Network (CNN). This approach allows for rapid dataset refinement with minimal manual effort. Experimental results on standard benchmarks demonstrate that while DS involves a performance trade-off compared to fully supervised methods, the proposed feedback loop provides critical insights into model generalization and significantly reduces the expert's workload.
Introduction: The Challenge of Relation Extraction
Relation Extraction (RE) involves identifying structured relationships between entities in text, such as "Steve Jobs founded Apple." While deep learning models have advanced the state-of-the-art, their reliance on massive amounts of labeled data remains a bottleneck. Distant Supervision (DS) offers a solution by automatically labeling sentences using existing knowledge bases. However, DS is inherently noisy, as many sentences containing two related entities do not actually express the target relationship.
As illustrated in Figure 2, there is often a significant gap between the knowledge captured by distant supervision and the absolute knowledge required for accurate relation extraction. The goal of this research is to bridge this gap by making efficient use of limited expert time to clean and refine distantly supervised datasets.
Methodology: Distant Supervision and the Expert-in-the-Loop
The researchers implemented a Ranking Convolutional Neural Network (CNN) and enhanced it with two primary strategies to handle the limitations of automated labeling.
First, Multi-instance Learning (MIL) was employed to manage the noise in DS. Instead of treating every auto-labeled sentence as ground truth, the model groups sentences containing a specific entity pair into a "bag." It assumes at least one sentence in the bag is correct and uses the highest-scoring example to update the model parameters.
Second, the paper introduces a trigram-based interpretability and feedback loop. The CNN identifies the most "representative trigrams" (three-word sequences) it uses to classify a relation. A domain expert then reviews these trigrams to validate the model's logic.
As shown in Figure 1, if the expert identifies a trigram that indicates overfitting (e.g., a specific entity name rather than a linguistic pattern), they can flag it. The system then filters out training sentences containing these "bad" trigrams and retrains the model, creating an iterative refinement process.
Experimental Setup and Results
The approach was evaluated using the SemEval 2010 Task 8 benchmark, the KBP37 dataset (18 directional relations from Wikipedia), and the New York Times Corpus combined with Wikidata.
Key findings include:
- Baseline Performance: The supervised Ranking CNN achieved an 84.39 F1 score on SemEval, surpassing previous RNN-based models.
- Impact of MIL: On the KBP37 dataset, the addition of Multi-instance Learning to the DS model improved the F1 score by approximately 2%, confirming its effectiveness in mitigating noise.
- Efficiency vs. Accuracy: While DS performed lower than pure supervised learning (approx. 45 F1 vs. 61 F1 on KBP37), the authors emphasize that the manual effort required for DS is zero. For many practical applications, this trade-off is acceptable given the high cost of manual annotation.
Discussion: The Overfitting Paradox
A significant insight from the study was the "cheating" behavior of the models. In many cases, the CNN memorized specific entity names (e.g., city names for a residency relation) rather than learning the underlying linguistic structure. Interestingly, removing these entity-specific trigrams sometimes lowered the F1 score on standard benchmarks. This "Overfitting Paradox" occurs because test sets often share entities with the training sets, rewarding models for memorization rather than true generalization.
Conclusion
This research demonstrates that combining distant supervision with a targeted expert feedback loop can significantly reduce the burden of data annotation in relation extraction. While representative trigrams are a relatively coarse tool for model interpretation, they provide a practical means for experts to guide model learning. Future work will focus on applying this workflow to highly specialized domains like medical text and developing more nuanced linguistic features for the feedback mechanism.
Summary updated