Introduction
In fields such as social science and journalism, researchers often utilize machine learning not to classify individual documents, but to estimate the proportion of a population that belongs to a specific category (e.g., the percentage of news articles using a specific policy frame). This task, known as quantification or proportion estimation, is frequently complicated by dataset shift, where the distribution of labels in the training (source) data differs from the target data.
Theoretical Framework: Intrinsic vs. Extrinsic Labels
Card and Smith argue that the choice of quantification method should depend on the nature of the labels. They introduce a critical distinction between two label types:
- Intrinsic Labels: The label is inherent to the document and causes the text (e.g., an author's identity determines their writing style). These are associated with prior probability shift.
- Extrinsic Labels: The text exists first, and a human annotator assigns a label based on content (e.g., sentiment or framing). These are associated with covariate shift, where the relationship between text and label ($p(y|x)$) remains constant, but the distribution of text ($p(x)$) changes.
Most existing literature relies on Adjusted Classify and Count (ACC), which assumes prior probability shift. However, the authors argue that for most human-annotated tasks (extrinsic labels), this assumption is fundamentally flawed.
Methodology: The PCC-cal Approach
To address extrinsic labels, the authors propose Calibrated Probabilistic Classify and Count (PCC-cal). While standard PCC averages predicted probabilities, PCC-cal specifically tunes model hyperparameters to minimize Calibration Error (CE) on held-out data rather than maximizing accuracy or F1-score. A well-calibrated model ensures that its predicted probabilities accurately reflect the true likelihood of a label, which is essential for aggregate statistics.
Experimental Results
The authors evaluated several methods across four datasets: Media Frames Corpus (MFC) and Amazon Reviews (Extrinsic), and Yelp and Twitter (Intrinsic).
Key Findings:
- Superiority of PCC-cal for Extrinsic Labels: On the MFC and Amazon datasets,
PCC-calsignificantly reduced Absolute Error (AE) compared to standard classifiers and the widely-used ACC method. - The Risk of ACC: While ACC performed well on intrinsic labels with artificial shifts, it often performed worse than a basic classifier on the messy, real-world shifts characteristic of social science data.
- The SRS Baseline: A sobering finding was that Simple Random Sampling (SRS)—manually labeling a small random subset of the target data—often outperformed complex machine learning models, regardless of the amount of training data available.
Conclusion
For researchers working with human-annotated text, the key to accurate proportion estimation is not a complex correction formula like ACC, but rather a well-calibrated classifier. Models that "know what they don't know" provide far more reliable aggregate statistics than models optimized solely for individual classification accuracy. Future work should explore how active learning can further improve calibration in high-dimensional text environments.
Summary updated