Introduction: The Challenge of Continuous Measurement
In the social sciences, researchers often need to measure variables that exist on a spectrum rather than in discrete categories. For instance, determining the intensity of emotional expression in a survey or the degree of negativity in a political advertisement requires a continuous (scalar) measurement. While Large Language Models (LLMs) are powerful text analyzers, they struggle with outputting reliable raw numbers.
When asked to rate a text on a 1–9 scale, LLMs frequently exhibit "heaping"—a phenomenon where responses cluster around specific, arbitrary numbers like 1, 5, or 9. This behavior, driven by biases in pre-training data, ruins the calibration of the scale and makes the relative distances between scores statistically meaningless.
Methodology: Four Approaches to Scoring
To address the calibration issue, this research evaluates four distinct strategies for extracting continuous measurements from LLMs:
- Pointwise Prompting (Direct Scoring): The standard approach of asking the LLM to output a direct score (e.g., 1 to 9).
- Token-Probability-Weighted Pointwise Scoring: Instead of taking the single most likely token, researchers extract the underlying probabilities for all numeric tokens (1-9) and calculate a weighted average. This "Averaged score" provides a much smoother and more calibrated distribution.
- Pairwise Comparisons: The model compares two texts and identifies which is more extreme. These comparisons are then aggregated using a Bradley-Terry model to infer latent scores.
- Finetuning (Reward Modeling): Training smaller, specialized models like DeBERTa-v3-large or Llama-3.1-8B on pairwise comparison data using a reward modeling objective.
Datasets and Evaluation
The study utilizes three datasets with ground-truth human pairwise annotations:
- Immigration Fear: 334 survey responses regarding anxiety about immigration.
- Ad-Negativity: 935 political campaign transcripts rated for opponent-directed negativity.
- Grandstanding: 3,499 congressional speeches rated for opinionated vs. factual content.
Key Insights and Results
The research yields several surprising findings regarding the behavior of LLMs versus humans. While humans are generally more reliable at pairwise comparisons than direct scoring, the probability-weighted direct scoring method actually proved most effective for LLMs. Using 5-shot pointwise prompting with this technique, Qwen-2.5-72B achieved high Spearman correlations ($\rho$) of 0.87, 0.92, and 0.66 across the datasets.
Furthermore, the study highlights that high correlation does not imply good calibration. Larger models like Llama-3.3-70B often rank items correctly but produce highly distorted absolute scores. Most importantly, the results show that small models punch above their weight. A DeBERTa-v3-large model finetuned on just 2,000 pairwise examples matched or exceeded the performance of 72B parameter models, offering a cost-effective and highly accurate alternative for researchers.
Conclusion and Future Directions
While these methods bring LLM performance close to human-level agreement, challenges remain. Accessing token probabilities is often restricted in closed APIs like GPT-4, limiting the use of the optimal weighted-averaging approach. Future research should focus on active sampling strategies to minimize the number of comparisons needed to build accurate scales and explore how these dynamics translate across different languages and cultural contexts.
Summary updated