TL;DR
When using Large Language Models (LLMs) to annotate text, we typically discard everything except the single most likely category. This paper demonstrates that retaining the full probability distribution ("soft labels") significantly improves the precision of downstream statistical analyses and can reduce human labeling requirements by up to 12% when using human-in-the-loop correction methods.
What Problem Does This Paper Address?
In fields like the social sciences, researchers use LLMs to label massive text datasets (e.g., classifying thousands of news sentences as "factual" vs. "opinionated") to perform downstream regression analysis. The standard practice is to use "hard labels"—the single category the model is most confident in.
However, language is often ambiguous, and category boundaries are fuzzy. A hard label hides whether the model was 99% certain or just barely preferred one category over another (e.g., a 51/49 split). This "hardening" step is an information-loss decision. The authors argue that this discarded uncertainty contains valuable signals that, if preserved, can make statistical estimates more accurate and efficient, especially when combined with a small amount of "gold standard" human-labeled data.
Methods
The researchers compared "soft" representations (the full probability vector) against "hard" representations (one-hot vectors of the top choice) across three different LLMs. They evaluated these representations within two primary human-assisted correction frameworks:
- Prediction-Powered Inference (PPI): A method that uses model predictions as an auxiliary signal to improve the estimation of human-defined targets.
- Design-Based Supervised Learning (DSL): A framework that "learns" a correction by training an auxiliary model (like a linear regression or random forest) on a small human-labeled sample to map LLM outputs to human labels.
The core innovation is demonstrating that soft labels serve as a much more informative "auxiliary measurement" than hard labels when anchoring automated counts to human judgments.
Datasets
The study utilized three social-science text datasets:
- BABE: 3,575 news sentences annotated by experts for media bias (Factual, Mixed, or Writer Opinion).
- Measuring Hate Speech (MHS): 2,974 social media comments labeled for hate speech.
- Crowd Counting Consortium (CCC): 4,271 news stories about protest events, categorized by event type (Protest, Rally, etc.).
Key Insights and Ideas
- LLM Uncertainty Mirrors Human Disagreement: The authors found that LLM probabilities correlate with human annotator agreement. Even among texts that receive the same "hard" label, those with higher LLM probability scores tend to have higher agreement among human voters (see Figure 1).
- Correction is the Catalyst: Interestingly, soft labels are not always better than hard labels for uncorrected regression. Their superiority consistently emerges only after human-labeled data is introduced to correct the model's systematic biases.
- Information Beyond the Argmax: The relative weights assigned to the "losing" categories are not just noise; they carry meaningful information about the human-coded outcome that hardening completely erases.
Key Results
- Consistent Efficiency Gains: Across all models and datasets, using soft labels with DSL (Soft+DSL) consistently outperformed hard-label DSL in terms of Root Mean Squared Error (RMSE) for regression coefficients.
- Significant Annotation Savings: To achieve the same level of statistical precision, researchers using soft labels required significantly fewer human annotations—saving between 5.7% and 11.9% of the labeling budget in the BABE and CCC datasets (see Figure 2).
- Better Distributional Alignment: Soft probability vectors showed a much lower Jensen-Shannon divergence from empirical human vote distributions compared to hard labels, indicating they capture the nuances of human judgment more effectively.
Limitations and Future Work
- Task Specificity: The study focused specifically on categorical annotation and regression-based downstream tasks. It remains to be seen if these gains hold for other types of analysis, like causal inference or discovery.
- Model Access: The method requires access to model probabilities (logprobs), which are increasingly restricted in some commercial "black box" LLM APIs.
- Reference Label Bias: Human-assisted correction can fix discrepancies between the LLM and the human reference, but it cannot correct for systematic biases inherent in the human labels themselves.
- Future Directions: The authors suggest exploring more complex "soft" representations and testing these methods across a broader range of domains beyond social science.
Summary updated