Introduction
Prevalence estimation, also known as quantification, is the task of determining the proportion of a specific class within a large, unlabeled population of documents. For example, a researcher might want to estimate the percentage of positive sentiment in millions of tweets regarding a new product. The most common approach, "Classify and Count" (CC), involves training a classifier on labeled data and aggregating its predictions on the unlabeled set.
However, CC is fundamentally flawed in two ways. First, it is highly sensitive to class prior shift, where the distribution of classes in the test set differs significantly from the training set. If a model trained on a balanced dataset is applied to a population that is 90% negative, its inherent biases will lead to severe estimation errors. Second, standard methods fail to quantify uncertainty. In scientific or industrial applications, a point estimate is often insufficient; understanding the model's confidence through statistically rigorous intervals is critical for valid inference.
Methodology: Bridging Generative and Discriminative Models
To address these issues, we propose a generative probabilistic framework that treats the group-level prevalence ($\theta$) as a latent variable. While generative models like Naive Bayes naturally handle uncertainty, they often lack the predictive accuracy of discriminative models like Logistic Regression in NLP tasks.
Our primary contribution, LR-Implicit, bridges this gap. We utilize an off-the-shelf Logistic Regression classifier and mathematically derive an implicit likelihood function. This allows us to integrate a high-performing discriminative model into a Bayesian generative framework. By calculating the marginal log-likelihood over the prevalence parameter $\theta$, the model can produce both a maximum likelihood point estimate and Bayesian credible intervals that reflect the true uncertainty of the estimation.
Experimental Setup
We evaluated our approach using the Yelp Dataset Challenge corpus, focusing on binary sentiment classification. To simulate realistic conditions, we created 500 distinct test groups, each representing a specific business with at least 200 reviews. We simulated a constrained annotation budget by training on only 2,000 labeled documents.
Two training scenarios were tested:
- Natural Prevalence: Training data reflected the natural Yelp distribution (~80% positive).
- Synthetic Shift: Training data was fixed at 10% positive to test robustness against extreme distribution shifts.
Key Results and Discussion
The experimental results demonstrate that LR-Implicit significantly outperforms traditional aggregation methods, particularly under distribution shift.
Robustness to Class Prior Shift
Under the synthetic shift scenario, the baseline CC method failed catastrophically, yielding a Mean Absolute Error (MAE) of 0.503. In contrast, LR-Implicit maintained high accuracy with an MAE of 0.069. This highlights the model's ability to "correct" for training biases when inferring group-level statistics.
Uncertainty Quantification
A critical finding was the failure of standard probabilistic baselines to provide reliable confidence intervals. The PB-PCC method exhibited extreme overconfidence, with its 90% confidence interval containing the true prevalence only 0.001% of the time under shift. LR-Implicit dramatically improved this coverage to 43.9%. While still short of the nominal 90% target—indicating remaining overconfidence in large groups—it represents a massive leap in statistical reliability for NLP prevalence estimation.
Conclusion
This work demonstrates that the statistical rigor of generative models can be combined with the predictive power of discriminative classifiers. By "backing out" implicit likelihoods, we can perform prevalence estimation that is both accurate under distribution shift and aware of its own uncertainty. Future research will explore extending this framework to modern neural architectures and multiclass settings.
Summary updated