Uncover the Ground-Truth Relations in Distant Supervision: A Neural Expectation-Maximization Framework

Junfan Chen, Richong Zhang, Yongyi Mao, Hongyu Guo, Jie Xu

Introduction

Relation Extraction (RE) is the task of identifying semantic relationships between entities in text, such as recognizing that "Larry Page" is employed_by "Google". To avoid the high cost of manual labeling, researchers use Distant Supervision (DS), which automatically creates training data by aligning a knowledge base with a text corpus.

However, DS is notoriously noisy. Most existing research focuses on "noisy sentences"—the idea that not every sentence containing two entities actually expresses the relationship. This paper highlights a deeper issue: noisy labels. Standard DS assumes that if a knowledge base says a relationship exists, that label is a "clean" target for training. In reality, these labels are often missing or incorrect. The authors argue that treating these "bag labels" as absolute ground truth leads to significant performance degradation.

The Neural Expectation-Maximization (nEM) Framework

The authors propose the nEM framework, which combines probabilistic graphical modeling with deep learning to treat ground-truth labels as latent variables.

Probabilistic Modeling

They model the process as a Markov chain: $X \to Y \to Z$.

  • $X$ is the input bag of sentences.
  • $Y$ is the latent ground-truth label.
  • $Z$ is the observed noisy label provided by distant supervision.

Neural Integration and EM Algorithm

Any standard neural architecture (like a Piecewise Convolutional Neural Network, or PCNN) can be "nested" within this framework to encode the bag semantics ($X$). To train the model, they use the Expectation-Maximization algorithm:

  • E-Step: Estimate the posterior probability of the latent ground-truth labels ($Y$) given the current model and the noisy observations ($Z$).
  • M-Step: Update the neural network parameters and the noise model parameters to maximize the likelihood of the data using stochastic gradient ascent.

Datasets and Evaluation

The paper evaluates the framework on two primary datasets:

  • Riedel (NYT) Dataset: The standard benchmark for distant supervision. The authors manually re-annotated 3,762 bags from the test set to create a reliable ground-truth evaluation set. They discovered that roughly 36% of labels in non-NA bags and 53% of labels in NA bags were originally incorrect.
  • TACRED: A large supervised dataset used to create a semi-synthetic DS dataset by injecting controlled amounts of artificial noise. This allowed precise measurement of how the nEM framework handles varying levels of label corruption.

Key Insights: The NSNL Assumption

A core contribution of this work is moving beyond simpler assumptions to the Noisy-Sentence Noisy-Label (NSNL) setting, which the authors argue is the most realistic representation of real-world data.

Comparison of noise settings in distant supervision: (a) Clean-Sentence Clean-Label, (b) Noisy-Sentence Clean-Label, (c) Clean-Sentence Noisy-Label, and (d) Noisy-Sentence Noisy-Label.

By treating the true label as latent, the model learns to "distrust" the distant supervision signal if the text semantics ($X$) strongly suggest a different relationship. The EM process acts as a sophisticated denoising mechanism. As training progresses, the model's internal estimate of the "true" label becomes more accurate, allowing it to ignore the "wrong" labels that would otherwise confuse a standard classifier.

Key Results

  • Performance Boost: On the Riedel dataset, nEM versions of standard models (PCNN+ATT, PCNN+MAX) consistently outperformed their baselines. The improvement was particularly dramatic in the low-recall regime, where precision stayed much higher than the baselines.
  • Robustness to Noise: In the TACRED experiments, as the injected noise level increased, the gap between nEM and the baseline PCNN widened. nEM showed a much stronger capability to maintain high recall despite corrupted training labels.
  • Predictive Confidence: Analysis showed that nEM produces much higher probability scores for correct labels compared to standard models, indicating it is more "confident" once it uncovers the ground truth.

Limitations and Future Work

The EM algorithm requires multiple updates in the M-step, which increases training time compared to standard end-to-end neural training. Additionally, the noise model parameters and the number of M-step updates require careful tuning. Future directions include developing more complex noise models to further improve the framework's ability to handle diverse types of label noise.

Summary updated

BibTeX
@inproceedings{chen2019uncover,
  author = {Junfan Chen and Richong Zhang and Yongyi Mao and Hongyu Guo and Jie Xu},
  title = {{Uncover the Ground-Truth Relations in Distant Supervision: A Neural Expectation-Maximization Framework}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1031},
}
Figure