Foreshadowing the Benefits of Incidental Supervision

Hangfeng He, Mingyuan Zhang, Qiang Ning, Dan Roth

Abstract

Modern NLP often relies on "incidental" supervision—imperfect signals like noisy labels, partial annotations, or cross-domain data—to supplement scarce high-quality "gold" data. However, selecting the most beneficial signals typically requires expensive trial-and-error. This paper introduces PABI (PAC-Bayesian motivated Informativeness), a unified mathematical measure designed to predict the utility of incidental supervision before training. By quantifying uncertainty reduction in the hypothesis space using information-theoretic principles, PABI achieves high correlation (0.92–0.96) with actual model performance improvements across tasks like Named Entity Recognition (NER) and Question Answering (QA).

Introduction

In the absence of sufficient gold-standard datasets, researchers frequently leverage cheap but abundant incidental supervision. These signals take various forms, including partial labels (incomplete tagging), noisy labels (distant supervision), logical constraints (e.g., BIO tagging rules), and cross-domain data.

Examples of incidental supervision signals in a Named Entity Recognition (NER) task.

Despite their availability, it is difficult to determine a priori which signals will most effectively improve model performance. Current practices rely on brute-force experimentation, which is computationally prohibitive. PABI addresses this by providing a predictive framework to identify the most helpful data sources without exhaustive training.

Methodology: PABI

PABI is grounded in PAC-Bayesian theory, viewing incidental signals as a means to refine a "prior" distribution over models into a more accurate "posterior."

Uncertainty Reduction

The core principle is that a useful signal should move the model's starting assumptions closer to the true "gold" distribution. PABI measures this shift using Kullback–Leibler (KL) divergence and entropy.

A Unified Metric

Unlike previous heuristics limited to specific noise types, PABI offers a "common currency" for all signals. It calculates how much an incidental signal shrinks the space of possible correct answers. For practical application where the gold distribution is unknown, the authors utilize an entropy-based approximation to estimate the informativeness of a signal relative to a state of zero knowledge.

Experimental Evaluation

The authors validated PABI using two primary NLP tasks:

  • Named Entity Recognition (NER): Using OntoNotes as the target, with CoNLL, Twitter, and GMB as cross-domain sources.
  • Question Answering (QA): Using SQuAD as the target, with incidental signals from QAMR, QA-SRL, QA-RE, NewsQA, and TriviaQA.

Key Results

The study found a remarkably strong positive correlation (Pearson coefficients between 0.92 and 0.96) between PABI scores and the actual relative improvement in model performance (F1 or Exact Match).

Correlation between PABI (Informativeness) and Relative Improvement across various types of incidental supervision.

PABI successfully distinguishes between different signal types, correctly predicting that certain partial labels are more informative than noisy ones even when error rates appear similar. In cross-domain scenarios, PABI significantly reduces the cost of model selection; instead of training $T^2$ joint models for $T$ domains, researchers only need $T+T$ models to calculate PABI and select the best source.

Conclusion

PABI provides a theoretically grounded and practically efficient method for evaluating incidental supervision. While current limitations include sensitivity to imbalanced label distributions and a simplified uniform noise assumption, PABI represents a significant step toward principled data selection in NLP. Future work will focus on handling instance-dependent noise and tightening the theoretical links between the measure and final error bounds for complex auxiliary tasks.

Summary updated

BibTeX
@inproceedings{he2021foreshadowing,
  author = {Hangfeng He and Mingyuan Zhang and Qiang Ning and Dan Roth},
  title = {{Foreshadowing the Benefits of Incidental Supervision}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.134},
}
Figure