Learning to Denoise Distantly-Labeled Data for Entity Typing

Yasumasa Onoe, Greg Durrett

Abstract

Distantly-labeled data can be used to scale up training of statistical models, but it is typically noisy and that noise can vary with the distant labeling technique. In this work, we propose a two-stage procedure for handling this type of data: denoise it with a learned model, then train our final model on clean and denoised distant data with standard supervised training. Our denoising approach consists of two parts. First, a filtering function discards examples from the distantly labeled data that are wholly unusable. Second, a relabeling function repairs noisy labels for the retained examples. Each of these components is a model trained on synthetically-noised examples generated from a small manually-labeled set. We investigate this approach on the ultra-fine entity typing task of Choi et al. (2018). Our baseline model is an extension of their model with pre-trained ELMo representations, which already achieves state-of-the-art performance. Adding distant data that has been denoised with our learned models gives further performance gains over this base model, outperforming models trained on raw distant data or heuristically-denoised distant data.

Introduction: The Noise in Distant Supervision

Deep learning models for Natural Language Processing (NLP) require vast amounts of labeled data. To meet this demand without the prohibitive cost of human annotation, researchers often turn to distant supervision. This technique leverages external knowledge bases (like Wikipedia or WordNet) or heuristics to automatically tag large corpora.

However, distant supervision is inherently "noisy." It typically suffers from two primary issues:

  1. False Positives: An entity mention may be incorrectly tagged due to ambiguity. For instance, "Gascoyne" might be labeled as a region based on a geographical location, even when the context clearly refers to a person.
  2. Incompleteness: Heuristics often miss valid labels. A span might be labeled as a film but lack related tags like movie, show, or creation.
Examples of noise in distantly-labeled data: (a, b, d) show incompleteness, while (c) shows a false positive.

In the context of Ultra-Fine Entity Typing, where models must choose from over 10,000 categories, these errors become a significant bottleneck. Naively incorporating this noisy data often degrades model performance.

The Denoising Framework

To address these issues, we propose a two-stage denoising pipeline that acts as a pre-processor for distant data before it is used to train the final typing model.

1. Filtering and Relabeling

The pipeline consists of two primary components:

  • Filtering Function ($f$): A binary classifier that evaluates a sentence and its automatically generated labels. If the labels are deemed "wholly unusable"—meaning they are entirely incorrect for the given context—the filter discards the example.
  • Relabeling Function ($g$): For examples that pass the filter, this component "repairs" the label set. It functions as a conditional denoising autoencoder, capable of adding missing labels or removing slightly incorrect ones to produce a cleaner set of tags.
The proposed denoising framework featuring filtering and relabeling components.

2. Training with Synthetic Noise

A key innovation of this approach is the training methodology for the denoisers. Because "clean" versions of the noisy distant data are unavailable, we utilize a small set of manually-labeled (gold) data and apply synthetic noise to simulate distant supervision errors:

  • To train the Filter, gold labels are replaced with random, non-overlapping labels.
  • To train the Relabeler, we randomly drop a significant portion (e.g., 70%) of the gold labels to mimic the incompleteness characteristic of distant supervision.

Model Architecture

The system employs a sophisticated Mention Encoder utilizing ELMo embeddings, character-level CNNs, and headword features. Complementing this is a Type Encoder that incorporates WordNet definitions. By encoding dictionary definitions, the model develops a semantic understanding of rare types, allowing it to generalize effectively across the vast "ultra-fine" label space.

Key Results

Our experiments on the Ultra-Fine Entity Typing dataset demonstrate the effectiveness of the denoising pipeline:

  • Performance Gains: The denoising framework provided a 3.0 F1 point improvement over a strong ELMo-based baseline.
  • Matching State-of-the-Art: The ELMo-based model with denoising achieved an F1 of 40.1, matching the performance of a fine-tuned BERT-Base model despite having fewer parameters.
  • Efficiency: On the OntoNotes dataset, we found that denoising just 2 million examples could nearly match the performance of other methods using 25 million raw, noisy examples. This underscores that data quality is often more impactful than sheer quantity.
  • Recall Improvement: The relabeling function was particularly effective at improving recall by successfully "filling in the blanks" for incomplete distant labels.

Conclusion

This work demonstrates that a learned denoising framework can effectively transform millions of noisy, automatically-labeled examples into a high-quality training resource. By simulating noise on a small gold-standard dataset, we can train robust filters and relabelers that significantly enhance the performance of entity typing models, reaching parity with much larger architectures like BERT.

Summary updated

Abstract

Distantly-labeled data can be used to scale up training of statistical models, but it is typically noisy and that noise can vary with the distant labeling technique. In this work, we propose a two-stage procedure for handling this type of data: denoise it with a learned model, then train our final model on clean and denoised distant data with standard supervised training. Our denoising approach consists of two parts. First, a filtering function discards examples from the distantly labeled data that are wholly unusable. Second, a relabeling function repairs noisy labels for the retained examples. Each of these components is a model trained on synthetically-noised examples generated from a small manually-labeled set. We investigate this approach on the ultra-fine entity typing task of Choi et al. (2018). Our baseline model is an extension of their model with pre-trained ELMo representations, which already achieves state-of-the-art performance. Adding distant data that has been denoised with our learned models gives further performance gains over this base model, outperforming models trained on raw distant data or heuristically-denoised distant data.
BibTeX
@inproceedings{onoe2019learning,
  author = {Yasumasa Onoe and Greg Durrett},
  title = {{Learning to Denoise Distantly-Labeled Data for Entity Typing}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=rkNLsfWObB},
}
Figure