TL;DR
This paper tackles the challenge of noisy labels in distant supervision for relation extraction by proposing a novel semantic label propagation method that leverages crowdsourced annotations. By propagating nuanced crowd insights to a large, noisy distant supervision dataset, they significantly improve relation classification performance with a much smaller crowdsourced dataset than previously thought necessary.
What Problem Does This Paper Address?
Relation extraction (RE) aims to identify semantic relationships between entities in text. A popular method for generating large-scale training data is "distant supervision" (DS), which assumes that if a knowledge base states a relation exists between two entities, any sentence containing those entities expresses that relation. However, this assumption leads to a significant number of false positives (noisy labels). While crowdsourcing has been used to correct these noisy labels, it's often resource-intensive. This paper addresses the gap of efficiently leveraging crowdsourced data to improve DS-based RE models, specifically by exploring how to propagate human-generated signals from a small, high-quality crowdsourced dataset to a much larger, noisy DS dataset.
Methods
The core of this paper is a semantic label propagation technique. Instead of relying on experts or complex dependency path analysis, they use the crowd to annotate actual sentence text. Their approach involves:
- Crowdsourcing with CrowdTruth: They utilize the CrowdTruth methodology, which captures annotation ambiguity by measuring inter-annotator disagreement. This results in nuanced "sentence-relation scores" (srs) rather than simple binary labels.
- Sentence Embedding: Sentences are represented as vectors by averaging word embeddings (specifically, word2vec) of the words between the entity pair. This focuses the representation on the most relevant parts of the sentence for relation expression.
- Similarity Calculation: Cosine similarity is used to measure the similarity between sentences in this embedding space.
- Label Propagation: For each sentence in the large DS corpus, they find the most similar sentence from the crowd-annotated set. The original DS label for that sentence is then updated by taking a weighted average of the original DS label and the crowd-derived srs of the most similar crowd-annotated sentence. The weight is determined by the cosine similarity between the two sentences. This effectively "propagates" the more reliable crowd signals to the noisy DS data.
Datasets
- CrowdTruth Corpus: A dataset of sentences with labels collected using the CrowdTruth methodology. This serves as the source of high-quality, nuanced annotations.
- Angeli et al. (2014) Corpus: Another dataset used to augment the CrowdTruth data.
- Freebase-derived DS Corpus: A large dataset of 235,000 sentences annotated by distant supervision, used for training the relation classifier.
The authors created a combined annotated dataset of 4,100 sentences (2,050 from CrowdTruth, 2,050 from Angeli et al.) for their experiments.
Key Insights and Ideas
- CrowdTruth for Nuance: The CrowdTruth methodology is crucial because it captures annotation ambiguity, providing richer signals (srs scores) than simple binary labels, which are more informative for propagation.
- Focusing Embeddings: Restricting sentence embeddings to words between the entity pair is an effective way to reduce noise and focus on the parts of the sentence most likely to express the relation.
- Scalability of Propagation: The semantic label propagation method is significantly more scalable than methods requiring expert annotators or complex feature engineering (like dependency paths), as it leverages the general crowd and standard word embeddings.
- Small Crowd Data is Powerful: The paper demonstrates that a relatively small crowdsourced dataset (2,050 sentences) can significantly improve performance when propagated to a much larger DS corpus, challenging the notion that massive crowdsourced datasets are always necessary.
Key Results
- The proposed DS + W2V CT method (DS data combined with semantic label propagation using word embeddings and CrowdTruth annotations) significantly outperforms the baseline DS model.
- Precision/Recall AUC: DS + W2V CT achieved a P/R AUC of 0.512, compared to 0.294 for DS alone.
- DS + CT (simply adding crowd-annotated sentences to DS data) also showed improvement (P/R AUC = 0.372) but was not as effective as the propagation method.
- The cosine similarity results (Figure 3) indicate that DS + W2V CT produces predictions closer to the crowd's agreement levels, suggesting better capture of relation confidence.
- The method achieved significant improvements using a crowd dataset that was two orders of magnitude smaller than the DS training set.
Limitations and Future Work
- Limitations: The paper acknowledges that some relations in the DS data have a very high false positive rate (e.g., origin, places_of_residence), indicating the inherent difficulty of correcting all errors with this method alone. The current approach relies on the quality of the initial DS data and the similarity of sentences in the embedding space.
- Future Work:
- Exploring different methods for collecting and aggregating crowd data, comparing approaches that focus on quality (multiple annotators per sentence, like CrowdTruth) versus quantity (one high-quality worker per sentence).
- Investigating whether disagreement-based metrics (like sentence and relation quality) can themselves be propagated.
- Applying this propagation method to correct training data for state-of-the-art relation extraction and knowledge-base population models.
Summary updated