Learning to Denoise Distantly-Labeled Data for Entity Typing

Yasumasa Onoe, Greg Durrett

Abstract: Yasumasa Onoe, Greg Durrett. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).

2019.

Introduction

Entity Typing is the task of assigning semantic categories (e.g., person, politician, artist) to entity mentions in text. To scale models to thousands of types, researchers rely on distant supervision, automatically labeling text using knowledge bases like Wikipedia. While efficient, this process introduces significant noise:

  1. Irrelevant Labels: Mis-linking entities (e.g., mistaking a person named "Gascoyne" for a region in Australia).
  2. Incomplete Labels: Missing specific tags (e.g., labeling "Rafael Nadal" as a player but missing tennis player or athlete).

Existing methods often attempt to handle this noise during training with complex probabilistic models. This paper proposes a two-stage denoising pipeline—filtering and relabeling—to clean distant data before it reaches the final classifier.

Methodology: The Denoising Pipeline

The core contribution is a modular denoising framework that treats data cleaning as a pre-processing step.

The proposed denoising pipeline consisting of a Filter and a Relabeler.

Filter and Relabeler

  • The Filter: A binary classifier that identifies and removes mentions where the distant labels are completely unrelated to the context.
  • The Relabeler: A conditional denoising autoencoder that "repairs" incomplete label sets. It takes noisy types and predicts a cleaner, more comprehensive set of labels that better matches human intuition.

Training with Synthetic Noise

To train these models without clean versions of the noisy data, the authors use a small set of manually labeled (gold) data. They synthetically inject noise—dropping or swapping labels—and train the Filter and Relabeler to reconstruct the original gold labels.

Architecture Highlights

The system uses ELMo embeddings and a "headword" feature to capture context. To handle over 10,000 types, the model encodes WordNet definitions of the types. This allows the model to leverage semantic relationships (e.g., understanding that a tennis player is an athlete) even for rare types.

Experimental Results

The approach was evaluated on the Ultra-Fine Entity Typing dataset (10,000+ types) and OntoNotes.

Performance comparison on the Ultra-Fine Entity Typing dataset.

Key findings include:

  • Performance Boost: Adding denoised distant data to the baseline ELMo model provided a 3.0 F1 gain on the Ultra-Fine task.
  • Competitive with BERT: The denoised ELMo model matched or outperformed a fine-tuned BERT-Base model, despite BERT having significantly more parameters and pre-training.
  • Recall Improvement: The Relabeler significantly boosted recall by filling in missing labels without a substantial loss in precision.
  • Generalization: On the OntoNotes dataset, the denoising method outperformed previous state-of-the-art models, proving the technique generalizes across different datasets.

Conclusion

By decoupling denoising from the final typing model, this work shows that "pre-washing" distant data is a highly effective strategy. Using label definitions and synthetic noise training allows the system to scale to massive, fine-grained type systems while maintaining high accuracy. Future work may explore integrating larger language models or applying this framework to other distantly supervised tasks like relation extraction.

Summary updated

BibTeX
@inproceedings{onoe2019learning,
  author = {Yasumasa Onoe and Greg Durrett},
  title = {{Learning to Denoise Distantly-Labeled Data for Entity Typing}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=rkNLsfWObB},
}
Figure