Hauke Licht, Rupak Sarkar, Patrick Y. Wu, Pranav Goel, Niklas Stoehr, Elliott Ash, Alexander Miserlis Hoyle.
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.
Introduction
In social science, many critical variables are not binary categories but scalar constructs—concepts that exist on a continuum. For instance, the degree of "populism" in a political speech or the "negativity" of a campaign advertisement requires nuanced measurement rather than simple classification. While Large Language Models (LLMs) offer a promising path for automating these measurements with minimal training data, they face a significant hurdle: "heaping."
When prompted to rate text on a numeric scale (e.g., 1–9), LLMs frequently cluster their responses around specific "favorite" numbers like 1, 5, or 8. This behavior creates "lumpy" distributions that fail to capture the underlying nuance of the text, leading to poor calibration compared to human judgment.
Methodological Strategies
To address these limitations, the researchers evaluated four distinct LLM-based scoring strategies across three political science datasets: Immigration Fear (survey responses), Ad-Negativity (campaign ads), and Grandstanding (congressional speeches).
- Direct Pointwise Scoring: Standard numeric rating (e.g., "Rate this 1–9").
- Pairwise Comparisons: Using a Bradley-Terry (BT) model to aggregate wins from head-to-head text comparisons into a latent scale.
- Token Probability-Weighted Scoring: Calculating an "expected value" by analyzing the probability distribution across all numeric tokens (1–9) rather than just the top-1 prediction.
- Finetuning: Training smaller encoder models (e.g., DeBERTa) specifically on pairwise comparison data using reward-modeling objectives.
Key Results and Performance
The study reveals that the "heaping" problem is a fundamental calibration issue rather than a prompting failure. However, for in-context learning, probability-weighted scoring effectively smooths these distributions, significantly improving alignment with human scores.
Furthermore, the results demonstrate that specialized smaller models can outperform general-purpose giants. A DeBERTa-v3-large model finetuned on as few as 1,000 pairs matched or exceeded the performance of Llama 3.3 70B. In the Grandstanding dataset, the finetuned model achieved a Spearman correlation of 0.78, compared to 0.65 for the best-prompted Llama model. This suggests that for specific social science constructs, data efficiency is high: with only 500–1,000 training pairs, small models become highly competitive.
Conclusion
For social scientists, the path to accurate scalar measurement lies in moving beyond simple zero-shot prompting. By utilizing token probabilities for off-the-shelf models or investing in small-scale finetuning, researchers can achieve high-precision measurements that are both computationally efficient and well-calibrated to human nuance. Future work will need to address the challenges of applying these techniques to closed-source models where token probabilities are often restricted.
Summary updated