Comparative Preference Scaling for Nuanced Human Judgments

47 concepts38 papers1 proposal

When researchers ask humans to quantify subjective qualities—such as the intensity of an emotion, the severity of a bias, or the relatedness of two words—they traditionally reach for Likert scales (e.g., "rate this from 1 to 7"). However, absolute scales are notoriously noisy. One annotator’s "7" might be another’s "5," and raters often struggle to maintain consistency over hundreds of items.

To solve this, a growing body of research across natural language processing, medicine, and psychology has turned to Best-Worst Scaling (BWS), also known as Maximum Difference Scaling (MaxDiff). Instead of asking for an absolute score, BWS asks annotators to make relative comparisons within small subsets of items. This shift from absolute to comparative judgment produces more reliable, fine-grained, and reproducible data.

The Cognitive Advantage of Relative Comparison

The core intuition behind BWS is that humans are significantly better at identifying extremes than they are at placing an item on an arbitrary numerical scale. As David Jurgens notes, absolute scale assignments carry a high cognitive difficulty, whereas identifying the "best" and "worst" options in a small set bypasses this burden.

In a typical BWS setup, an annotator is presented with a subset of items (often four) and asked to pick only two: the one that most represents the attribute in question and the one that least represents it. Saif M. Mohammad describes this as "comparative extreme selection." By forcing these trade-offs, the method reduces common biases, such as the tendency for raters to mark every item as highly important (Daniel et al.).

Because the subsets overlap across the entire dataset, these simple "best" and "worst" choices can be mathematically transformed into a continuous, real-valued ranking. This process yields a gold standard that is often more stable than traditional ratings (Asaadi et al.).

Quantifying the Fuzziness of Language

BWS has proven particularly effective for linguistic tasks where boundaries are inherently "fuzzy." For example, Jurgens demonstrated that untrained crowdsourced workers could match expert quality in word sense annotation by using BWS to "embrace ambiguity" rather than being forced into binary choices.

The method has been used to map out several nuanced linguistic dimensions:

  • Sentiment and Emotion: Rosenthal et al. and Salameh et al. used BWS to move beyond "positive vs. negative" classification toward real-valued intensity scores for words and phrases.
  • Social Dynamics: Jurgens and Pei applied the technique to quantify "intimacy," a latent social variable that is difficult to capture with discrete labels.
  • Semantic Nuance: In the BiRD dataset, Asaadi et al. used BWS to show that in phrases like "apple pie," the modifier often carries more semantic weight than the head noun.

Distinguishing High-Quality Model Outputs

As Large Language Models (LLMs) improve, the differences between their outputs become increasingly subtle. Traditional rating scales often fail to distinguish between two summaries that are both "good" but differ in factual consistency.

Tang et al. found that BWS is more reliable than Likert scales for evaluating factual consistency in summaries because it forces a discriminative choice between similar items. Similarly, Hada et al. used BWS to create a normative dataset of gender bias severity, finding that while LLMs can identify bias, they struggle to quantify its intensity as consistently as the BWS-derived human gold standard.

Applications Beyond NLP

The utility of BWS extends into high-stakes decision-making where understanding trade-offs is critical. In healthcare, Meng et al. used a "multiprofile" BWS experiment to show that cancer patients prioritize quality of life over the mere length of survival. In clinical practice, Yan et al. used the method to reveal that gastroenterologists are often driven by the "visual confirmation" of a procedure's success rather than just clinical guidelines.

The Cost of Precision

While BWS provides superior reliability, it is not without costs. Chu et al. observed that BWS can be significantly slower—up to 4.5 times—than categorical labeling. However, for researchers who require a stable, continuous scale for subjective attributes, the reduction in inter-annotator variance and the increase in data reproducibility (Arvan and Parde) often justify the additional effort.

Key Foundational Work

SemEval-2012 Task 2: Measuring Degrees of Relational Similarity (2012) — contributed 1 concept elements to this direction.
Embracing Ambiguity: A Comparison of Annotation Methodologies for Crowdsourcing Word Sense Labels (2013) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_best_worst_scaling_108b0f39,
  title = {best-worst scaling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/best-worst-scaling/rdr_108b0f39567f439e9bf720d40c604cf5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_best_worst_scaling_108b0f39_v1,
  title = {best-worst scaling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/best-worst-scaling/rdr_108b0f39567f439e9bf720d40c604cf5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake