Measuring Scalar Constructs in Social Science with LLMs

Hauke Licht, Rupak Sarkar, Patrick Y. Wu, Pranav Goel, Niklas Stoehr, Elliott Ash, Alexander Miserlis HoylearXiv 2025 · DOI · arXiv

Introduction: The Challenge of Continuous Measurement

In the social sciences, researchers often need to measure variables that exist on a spectrum rather than in discrete categories. For instance, determining the intensity of emotional expression in a survey or the degree of negativity in a political advertisement requires a continuous (scalar) measurement. While Large Language Models (LLMs) are powerful text analyzers, they struggle with outputting reliable raw numbers.

When asked to rate a text on a 1–9 scale, LLMs frequently exhibit "heaping"—a phenomenon where responses cluster around specific, arbitrary numbers like 1, 5, or 9. This behavior, driven by biases in pre-training data, ruins the calibration of the scale and makes the relative distances between scores statistically meaningless.

Figure 1: Comparison of human BT scores and LLM response distributions, highlighting the 'heaping' phenomenon in raw LLM outputs.

Methodology: Four Approaches to Scoring

To address the calibration issue, this research evaluates four distinct strategies for extracting continuous measurements from LLMs:

  1. Pointwise Prompting (Direct Scoring): The standard approach of asking the LLM to output a direct score (e.g., 1 to 9).
  2. Token-Probability-Weighted Pointwise Scoring: Instead of taking the single most likely token, researchers extract the underlying probabilities for all numeric tokens (1-9) and calculate a weighted average. This "Averaged score" provides a much smoother and more calibrated distribution.
  3. Pairwise Comparisons: The model compares two texts and identifies which is more extreme. These comparisons are then aggregated using a Bradley-Terry model to infer latent scores.
  4. Finetuning (Reward Modeling): Training smaller, specialized models like DeBERTa-v3-large or Llama-3.1-8B on pairwise comparison data using a reward modeling objective.
Figure 4: Analysis of response proportions, confidence, and averaged scores for Llama models, showing how probability weighting smooths the output distribution.

Datasets and Evaluation

The study utilizes three datasets with ground-truth human pairwise annotations:

  • Immigration Fear: 334 survey responses regarding anxiety about immigration.
  • Ad-Negativity: 935 political campaign transcripts rated for opponent-directed negativity.
  • Grandstanding: 3,499 congressional speeches rated for opinionated vs. factual content.

Key Insights and Results

The research yields several surprising findings regarding the behavior of LLMs versus humans. While humans are generally more reliable at pairwise comparisons than direct scoring, the probability-weighted direct scoring method actually proved most effective for LLMs. Using 5-shot pointwise prompting with this technique, Qwen-2.5-72B achieved high Spearman correlations ($\rho$) of 0.87, 0.92, and 0.66 across the datasets.

Furthermore, the study highlights that high correlation does not imply good calibration. Larger models like Llama-3.3-70B often rank items correctly but produce highly distorted absolute scores. Most importantly, the results show that small models punch above their weight. A DeBERTa-v3-large model finetuned on just 2,000 pairwise examples matched or exceeded the performance of 72B parameter models, offering a cost-effective and highly accurate alternative for researchers.

Conclusion and Future Directions

While these methods bring LLM performance close to human-level agreement, challenges remain. Accessing token probabilities is often restricted in closed APIs like GPT-4, limiting the use of the optimal weighted-averaging approach. Future research should focus on active sampling strategies to minimize the number of comparisons needed to build accurate scales and explore how these dynamics translate across different languages and cultural contexts.

Summary updated

Abstract

Many constructs that characterize language, like its complexity or emotionality, have a naturally continuous semantic structure; a public speech is not just "simple" or "complex," but exists on a continuum between extremes. Although large language models (LLMs) are an attractive tool for measuring scalar constructs, their idiosyncratic treatment of numerical outputs raises questions of how to best apply them. We address these questions with a comprehensive evaluation of LLM-based approaches to scalar construct measurement in social science. Using multiple datasets sourced from the political science literature, we evaluate four approaches: unweighted direct pointwise scoring, aggregation of pairwise comparisons, token-probability-weighted pointwise scoring, and finetuning. Our study finds that pairwise comparisons made by LLMs produce better measurements than simply prompting the LLM to directly output the scores, which suffers from bunching around arbitrary numbers. However, taking the weighted mean over the token probability of scores further improves the measurements over the two previous approaches. Finally, finetuning smaller models with as few as 1,000 training pairs can match or exceed the performance of prompted LLMs.
BibTeX
@misc{licht2025measuring,
  author = {Hauke Licht and Rupak Sarkar and Patrick Y. Wu and Pranav Goel and Niklas Stoehr and Elliott Ash and Alexander Miserlis Hoyle},
  title = {{Measuring Scalar Constructs in Social Science with LLMs}},
  year = {2025},
  eprint = {2509.03116},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.03116},
}
Figure