TL;DR
This paper introduces H-FND, a novel hierarchical denoising framework designed to tackle the pervasive issue of false negatives (FNs) in relation extraction datasets generated through distant supervision. By intelligently deciding whether to keep, discard, or revise potentially mislabeled negative instances, H-FND significantly improves the robustness and performance of relation extraction models, even in the presence of a high proportion of noisy data.
What Problem Does This Paper Address?
Relation extraction (RE) is a crucial task in information extraction, aiming to identify relationships between entities in text. Distant supervision (DS) is a popular technique for automatically generating large-scale training data by aligning knowledge bases with text corpora. However, DS inherently introduces noise in the form of false positives (FPs) and false negatives (FNs). While much prior work has focused on mitigating FPs, the problem of FNs has been relatively under-addressed, particularly for deep neural networks. FNs, where a true relation is incorrectly labeled as non-existent, can severely degrade model performance by reducing recall and leading to missed positive instances. This paper aims to fill this gap by proposing a method specifically designed to denoise FNs in DS datasets.
Methods
The core of H-FND is a hierarchical denoising policy that operates in two steps. First, a denoising agent, trained using policy-based reinforcement learning, makes a ternary decision for each negative instance:
- Keep: The instance remains labeled as negative.
- Discard: The instance is removed to prevent it from misleading the model.
- Revise: The instance is relabeled as a positive instance with a predicted relation.
If an instance is marked for revision, a relation classifier then predicts a new, more appropriate relation for it. This hierarchical approach is crucial because simply discarding FNs might lead to the loss of valuable training data, and revising them without knowing the correct relation type is insufficient.
To train the denoising agent, which lacks direct supervision, the authors propose a special transfer learning scheme. This scheme leverages the knowledge learned by a pre-trained relation classifier to generate "action labels" for the denoising agent. The relation classifier is first pre-trained on the original noisy dataset. Then, its predictions are used to label instances as "Revise" (if the classifier incorrectly predicts a relation for a negative instance), "Keep" (if it correctly predicts it as negative), or "Discard" (if it wrongly predicts a positive relation for a negative instance).
The overall framework operates in a co-training manner, where the denoising agent and relation classifier are trained iteratively. Each epoch involves the denoising agent making decisions, the relation classifier revising instances, and then the relation classifier being retrained on the "cleaned" dataset. The F1 score on a validation set is used as a reward signal to update the denoising agent's policy.
Datasets
The paper evaluates H-FND on the following datasets:
- SemEval-2010: A human-annotated dataset with nine relations and an additional "NA" (non-relation) class. For experiments, synthetic FNs were introduced by randomly flipping a percentage of positive instances into negatives.
- TACRED: A larger human-annotated dataset with 42 relations, where negative instances constitute about 80% of the corpus. Similar to SemEval, synthetic FNs were generated.
- NYT10: A distantly supervised dataset derived from the New York Times corpus and Freebase. This dataset was used to evaluate H-FND in a more realistic, less controlled setting.
The authors also created controlled FN settings for SemEval and TACRED by randomly flipping a ratio (10%-50%) of training/validation positives into negatives, while keeping the testing sets clean.
Key Insights and Ideas
- FNs are a critical bottleneck: The paper strongly demonstrates that false negatives significantly degrade relation extraction performance, often more so than false positives in certain scenarios, highlighting the need for dedicated FN denoising.
- Hierarchical denoising is key: A simple binary decision (keep/discard or keep/revise) is insufficient for FNs. The proposed three-way decision (Keep, Discard, Revise) combined with a subsequent relation prediction for revisions is more effective.
- Transfer learning for denoising agents: The novel transfer learning scheme to pretrain the denoising agent is a significant contribution, as it provides a way to train the agent without direct supervision, leveraging the relation classifier's learned knowledge.
- Robustness across FN ratios: H-FND shows remarkable resilience, maintaining high F1 scores even when up to 50% of instances are synthetically turned into false negatives, outperforming many existing denoising methods in these challenging conditions.
- No explicit noise rate needed: Unlike some other denoising methods (e.g., cleanlab), H-FND does not require prior knowledge of the noise rate, making it more practical for real-world applications where this information is often unknown.
Key Results
- On SemEval and TACRED with controlled FN ratios (up to 50%), H-FND consistently achieved higher F1 scores compared to baseline models and other denoising methods like IRMIE and co-teaching, especially at higher FN ratios. For instance, at a 50% FN ratio on SemEval, H-FND's F1 score was significantly better than the base model, which dropped by nearly 20%.
- On TACRED, H-FND maintained competitive performance even when the FN ratio exceeded 30%, and showed significant advantages at higher ratios (up to 90%), where other methods' performance dropped sharply.
- Experiments on the realistic NYT10 dataset showed that H-FND is applicable and effective in a real-world distant supervision setting, achieving strong precision-recall curves.
- Ablation studies confirmed the effectiveness of both the "Revise" action and the pretraining strategy, particularly on the more complex TACRED dataset.
Limitations and Future Work
- Simultaneous FP and FN denoising: The paper primarily focuses on FNs. The authors acknowledge that in realistic DS settings, both FPs and FNs can emerge simultaneously, and addressing both concurrently is a direction for future work.
- Advanced RE architectures: The authors suggest exploring the integration of H-FND with more advanced relation classification architectures, such as R-BERT, to further enhance performance.
- Human evaluation on NYT10: While H-FND was evaluated on NYT10, the paper mentions conducting human evaluation on a sample of negative instances, suggesting that further human validation could be beneficial.
- Noise rate estimation: While H-FND doesn't require a known noise rate, the paper notes that other methods like cleanlab perform similarly when provided with the correct rate, implying that accurate noise rate estimation could be a complementary approach.
Summary updated