Beyond Hard Labels: LLM Probabilities Improve Human-Assisted Regression

Xinyue Wang, Qunhua LiarXiv 2026 · arXiv

TL;DR

When using Large Language Models (LLMs) to annotate text, we typically discard everything except the single most likely category. This paper demonstrates that retaining the full probability distribution ("soft labels") significantly improves the precision of downstream statistical analyses and can reduce human labeling requirements by up to 12% when using human-in-the-loop correction methods.

What Problem Does This Paper Address?

In fields like the social sciences, researchers use LLMs to label massive text datasets (e.g., classifying thousands of news sentences as "factual" vs. "opinionated") to perform downstream regression analysis. The standard practice is to use "hard labels"—the single category the model is most confident in.

However, language is often ambiguous, and category boundaries are fuzzy. A hard label hides whether the model was 99% certain or just barely preferred one category over another (e.g., a 51/49 split). This "hardening" step is an information-loss decision. The authors argue that this discarded uncertainty contains valuable signals that, if preserved, can make statistical estimates more accurate and efficient, especially when combined with a small amount of "gold standard" human-labeled data.

Methods

The researchers compared "soft" representations (the full probability vector) against "hard" representations (one-hot vectors of the top choice) across three different LLMs. They evaluated these representations within two primary human-assisted correction frameworks:

  • Prediction-Powered Inference (PPI): A method that uses model predictions as an auxiliary signal to improve the estimation of human-defined targets.
  • Design-Based Supervised Learning (DSL): A framework that "learns" a correction by training an auxiliary model (like a linear regression or random forest) on a small human-labeled sample to map LLM outputs to human labels.

The core innovation is demonstrating that soft labels serve as a much more informative "auxiliary measurement" than hard labels when anchoring automated counts to human judgments.

Datasets

The study utilized three social-science text datasets:

  • BABE: 3,575 news sentences annotated by experts for media bias (Factual, Mixed, or Writer Opinion).
  • Measuring Hate Speech (MHS): 2,974 social media comments labeled for hate speech.
  • Crowd Counting Consortium (CCC): 4,271 news stories about protest events, categorized by event type (Protest, Rally, etc.).

Key Insights and Ideas

  • LLM Uncertainty Mirrors Human Disagreement: The authors found that LLM probabilities correlate with human annotator agreement. Even among texts that receive the same "hard" label, those with higher LLM probability scores tend to have higher agreement among human voters (see Figure 1).
  • Correction is the Catalyst: Interestingly, soft labels are not always better than hard labels for uncorrected regression. Their superiority consistently emerges only after human-labeled data is introduced to correct the model's systematic biases.
  • Information Beyond the Argmax: The relative weights assigned to the "losing" categories are not just noise; they carry meaningful information about the human-coded outcome that hardening completely erases.
Figure 1: Human vote alignment within fixed LLM hard predictions in BABE. Within each predicted category, the x-axis bins pmax and the y-axis gives the mean frac- tion of five human annotators selecting that category. Open markers denote bins with fewer than 20 texts.

Key Results

  • Consistent Efficiency Gains: Across all models and datasets, using soft labels with DSL (Soft+DSL) consistently outperformed hard-label DSL in terms of Root Mean Squared Error (RMSE) for regression coefficients.
  • Significant Annotation Savings: To achieve the same level of statistical precision, researchers using soft labels required significantly fewer human annotations—saving between 5.7% and 11.9% of the labeling budget in the BABE and CCC datasets (see Figure 2).
  • Better Distributional Alignment: Soft probability vectors showed a much lower Jensen-Shannon divergence from empirical human vote distributions compared to hard labels, indicating they capture the nuances of human judgment more effectively.
Figure 2: Annotation savings for Soft+DSL relative to DSL at matched aggregate coefficient RMSE. Positive values favor soft inputs; error bars are 95% Monte Carlo intervals. See Appendix D.4 for construction details.

Limitations and Future Work

  • Task Specificity: The study focused specifically on categorical annotation and regression-based downstream tasks. It remains to be seen if these gains hold for other types of analysis, like causal inference or discovery.
  • Model Access: The method requires access to model probabilities (logprobs), which are increasingly restricted in some commercial "black box" LLM APIs.
  • Reference Label Bias: Human-assisted correction can fix discrepancies between the LLM and the human reference, but it cannot correct for systematic biases inherent in the human labels themselves.
  • Future Directions: The authors suggest exploring more complex "soft" representations and testing these methods across a broader range of domains beyond social science.

Summary updated

Abstract

Large language models (LLMs) are increasingly used for text annotation, but their outputs are often reduced to a single categorical label before downstream analysis. This hardening step discards the model's relative support across categories, which may contain useful information when category boundaries are ambiguous. We study whether retaining the full probability vector improves downstream regression when LLM annotations are combined with a limited amount of human annotation. Using matched soft and hard representations from three LLMs across three social-science text datasets, we compare direct regression and human-assisted correction. Soft inputs are not uniformly better when used directly as outcomes, but after human correction, using the probability vector consistently improves regression efficiency across models and datasets. Analyses of repeated human judgments and controlled simulations further show that these gains arise when probability variation beyond the hard category carries useful information about the human-coded outcome. Our results identify output representation as an important design choice in LLM-assisted measurement and support preserving probability outputs for downstream statistical analysis. Code and reproducibility materials are available at https://anonymous.4open.science/r/soft-label-LLM-D201/
BibTeX
@misc{wang2026beyond,
  author = {Xinyue Wang and Qunhua Li},
  title = {{Beyond Hard Labels: LLM Probabilities Improve Human-Assisted Regression}},
  year = {2026},
  eprint = {2611.11111},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11111},
}
Figure