Abstract
In the absence of large-scale "gold standard" labeled data, machine learning practitioners often turn to incidental supervision—cheap but imperfect signals such as noisy labels, partial annotations, or cross-domain data. However, there has been no principled way to predict which of these signals will most effectively improve model performance without conducting exhaustive training experiments. We present PABI (PAC-Bayesian motivated Informativeness), a unified theoretical measure that quantifies the potential benefit of an incidental signal by measuring its "uncertainty reduction." Our evaluation across Named Entity Recognition (NER) and Question Answering (QA) tasks shows that PABI scores correlate strongly (Pearson and Spearman coefficients > 0.90) with actual empirical improvements, offering a computationally efficient alternative to trial-and-error data selection.
Introduction
The "gold standard" of supervised learning—massive amounts of perfectly labeled data—is often a luxury. In practice, we rely on incidental supervision: signals related to the target task but lacking perfection. These include:
- Partial labels: Only a subset of tokens or instances are annotated.
- Noisy labels: Data generated via distant supervision or weak heuristics.
- Constraints: Logical or structural rules (e.g., BIO tagging consistency).
- Cross-domain data: Leveraging datasets from related domains (e.g., news to social media).
Comparing these signals is traditionally difficult. If a practitioner has one noisy dataset and one partially labeled dataset, the current best practice is to "try both and see." This approach is computationally expensive and offers little theoretical insight.
Methodology: The PABI Measure
The authors introduce PABI, a measure derived from PAC-Bayesian theory. PAC-Bayesian frameworks provide mathematical bounds on generalization by considering the divergence between a prior belief and a posterior distribution after seeing data.
PABI quantifies uncertainty reduction: how much an incidental signal moves our "prior" belief closer to the "gold posterior" (the ideal knowledge state for the task). While the full theoretical derivation involves complex KL-divergence, the authors provide a practical entropy-based approximation:
$$\hat{S} = \sqrt{1 - \frac{H(\tilde{\pi}_0)}{H(\pi_0)}}$$
Here, $H(\tilde{\pi}_0)$ represents the entropy (uncertainty) of the incidental signal, and $H(\pi_0)$ is the entropy of a completely uninformed uniform distribution. A higher PABI score indicates greater entropy reduction and, theoretically, a more helpful supervision signal.
Experimental Evaluation
The framework was validated using two core NLP tasks:
- Named Entity Recognition (NER): Using Ontonotes as the target, with CoNLL, Twitter, and GMB as cross-domain incidental sources.
- Question Answering (QA): Using SQuAD as the target, with QAMR, QA-SRL, QA-RE, NewsQA, and TriviaQA as incidental sources.
Key Results and Insights
The primary finding is a strong positive correlation (often exceeding 0.90) between PABI scores and the actual relative improvement in model metrics (F1 score or Exact Match).
Unified Comparison
PABI serves as a "unified yardstick," allowing for "apples-to-oranges" comparisons. It can rank a set of logical constraints against a noisy dataset from a different domain on a single 0-to-1 scale.
Computational Efficiency
Calculating PABI is significantly faster than joint training. For a scenario with $T$ source and $T$ target domains, joint training requires $O(T^2)$ experiments, whereas PABI requires only $O(2T)$ evaluations.
Identifying Valueless Signals
PABI can identify when a signal provides little to no value. For instance, the authors found that common heuristics like vocabulary overlap are often poor predictors of model improvement, whereas PABI remains reliable.
Limitations and Future Directions
While robust, PABI faces certain challenges:
- Label Imbalance: In tasks like NER, where the "O" (Outside) tag dominates (~88%), the relative improvement can be smaller than PABI predicts.
- Concept Space: The current model assumes a finite set of labels, which may need adaptation for continuous output spaces.
- Noise Complexity: Future work will address "input-dependent" noise, where errors are correlated with specific sentence structures or features.
Conclusion
PABI bridges the gap between theoretical PAC-Bayesian bounds and practical machine learning engineering. By providing a reliable, efficient way to foresee the benefits of incidental supervision, it empowers practitioners to make data-driven decisions about which "cheap" signals are truly worth their time.
Summary updated