Foreseeing the Benefits of Incidental Supervision

Hangfeng He, Mingyuan Zhang, Qiang Ning, Dan Roth

Abstract

In the absence of large-scale "gold standard" labeled data, machine learning practitioners often turn to incidental supervision—cheap but imperfect signals such as noisy labels, partial annotations, or cross-domain data. However, there has been no principled way to predict which of these signals will most effectively improve model performance without conducting exhaustive training experiments. We present PABI (PAC-Bayesian motivated Informativeness), a unified theoretical measure that quantifies the potential benefit of an incidental signal by measuring its "uncertainty reduction." Our evaluation across Named Entity Recognition (NER) and Question Answering (QA) tasks shows that PABI scores correlate strongly (Pearson and Spearman coefficients > 0.90) with actual empirical improvements, offering a computationally efficient alternative to trial-and-error data selection.

Introduction

The "gold standard" of supervised learning—massive amounts of perfectly labeled data—is often a luxury. In practice, we rely on incidental supervision: signals related to the target task but lacking perfection. These include:

  • Partial labels: Only a subset of tokens or instances are annotated.
  • Noisy labels: Data generated via distant supervision or weak heuristics.
  • Constraints: Logical or structural rules (e.g., BIO tagging consistency).
  • Cross-domain data: Leveraging datasets from related domains (e.g., news to social media).
Visualizing different types of incidental supervision signals for a Named Entity Recognition (NER) task.

Comparing these signals is traditionally difficult. If a practitioner has one noisy dataset and one partially labeled dataset, the current best practice is to "try both and see." This approach is computationally expensive and offers little theoretical insight.

Methodology: The PABI Measure

The authors introduce PABI, a measure derived from PAC-Bayesian theory. PAC-Bayesian frameworks provide mathematical bounds on generalization by considering the divergence between a prior belief and a posterior distribution after seeing data.

PABI quantifies uncertainty reduction: how much an incidental signal moves our "prior" belief closer to the "gold posterior" (the ideal knowledge state for the task). While the full theoretical derivation involves complex KL-divergence, the authors provide a practical entropy-based approximation:

$$\hat{S} = \sqrt{1 - \frac{H(\tilde{\pi}_0)}{H(\pi_0)}}$$

Here, $H(\tilde{\pi}_0)$ represents the entropy (uncertainty) of the incidental signal, and $H(\pi_0)$ is the entropy of a completely uninformed uniform distribution. A higher PABI score indicates greater entropy reduction and, theoretically, a more helpful supervision signal.

Experimental Evaluation

The framework was validated using two core NLP tasks:

  1. Named Entity Recognition (NER): Using Ontonotes as the target, with CoNLL, Twitter, and GMB as cross-domain incidental sources.
  2. Question Answering (QA): Using SQuAD as the target, with QAMR, QA-SRL, QA-RE, NewsQA, and TriviaQA as incidental sources.

Key Results and Insights

The primary finding is a strong positive correlation (often exceeding 0.90) between PABI scores and the actual relative improvement in model metrics (F1 score or Exact Match).

Correlation between PABI-based informativeness and actual relative improvement across various supervision scenarios, including partial labels, noisy labels, and their combinations.

Unified Comparison

PABI serves as a "unified yardstick," allowing for "apples-to-oranges" comparisons. It can rank a set of logical constraints against a noisy dataset from a different domain on a single 0-to-1 scale.

Computational Efficiency

Calculating PABI is significantly faster than joint training. For a scenario with $T$ source and $T$ target domains, joint training requires $O(T^2)$ experiments, whereas PABI requires only $O(2T)$ evaluations.

Identifying Valueless Signals

PABI can identify when a signal provides little to no value. For instance, the authors found that common heuristics like vocabulary overlap are often poor predictors of model improvement, whereas PABI remains reliable.

Limitations and Future Directions

While robust, PABI faces certain challenges:

  • Label Imbalance: In tasks like NER, where the "O" (Outside) tag dominates (~88%), the relative improvement can be smaller than PABI predicts.
  • Concept Space: The current model assumes a finite set of labels, which may need adaptation for continuous output spaces.
  • Noise Complexity: Future work will address "input-dependent" noise, where errors are correlated with specific sentence structures or features.

Conclusion

PABI bridges the gap between theoretical PAC-Bayesian bounds and practical machine learning engineering. By providing a reliable, efficient way to foresee the benefits of incidental supervision, it empowers practitioners to make data-driven decisions about which "cheap" signals are truly worth their time.

Summary updated

Abstract

Real-world applications often require improved models by leveraging a range of cheap incidental supervision signals. These could include partial labels, noisy labels, knowledge-based constraints, and cross-domain or cross-task annotations -- all having statistical associations with gold annotations but not exactly the same. However, we currently lack a principled way to measure the benefits of these signals to a given target task, and the common practice of evaluating these benefits is through exhaustive experiments with various models and hyperparameters. This paper studies whether we can, in a single framework, quantify the benefits of various types of incidental signals for a given target task without going through combinatorial experiments. We propose a unified PAC-Bayesian motivated informativeness measure, PABI, that characterizes the uncertainty reduction provided by incidental supervision signals. We demonstrate PABI's effectiveness by quantifying the value added by various types of incidental signals to sequence tagging tasks. Experiments on named entity recognition (NER) and question answering (QA) show that PABI's predictions correlate well with learning performance, providing a promising way to determine, ahead of learning, which supervision signals would be beneficial.
BibTeX
@inproceedings{he2021foreseeing,
  author = {Hangfeng He and Mingyuan Zhang and Qiang Ning and Dan Roth},
  title = {{Foreseeing the Benefits of Incidental Supervision}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.134},
}
Figure