In natural language processing, we often ask humans to label data. For simple tasks like "is this word a noun?", the answer is binary. But for subjective qualities—how biased is this sentence? how intimate is this conversation? how related are these two phrases?—absolute ratings fail us.
Traditional approaches rely on Likert scales (e.g., rating intensity from 1 to 5). However, humans are notoriously inconsistent at assigning absolute values to abstract concepts. One person’s "4" is another’s "3," and raters often suffer from "scale fatigue," where they stop distinguishing between subtle differences. To solve this, researchers have converged on a comparative mechanism: Best-Worst Scaling (BWS).
The Problem with Absolute Ratings
When David Jurgens, Saif M. Mohammad, and colleagues introduced the SemEval-2012 Task 2: Measuring Degrees of Relational Similarity, they highlighted a core challenge: machines struggle to quantify the "graded" differences that make some analogies better than others.
The issue isn't just a machine learning problem; it's an annotation problem. Absolute scales are cognitively demanding because they require an internal, stable reference for what a "2" or a "5" means across thousands of examples. This leads to inter-annotator inconsistency and high variance.
The Comparative Intuition
BWS, also known as Maximum Difference Scaling, transforms the task of rating into a task of choosing. Instead of looking at one item in isolation, an annotator is presented with a small subset (usually 4 or 5 items) and asked only two questions:
- Which item is the most [X]?
- Which item is the least [X]?
As Saif M. Mohammad demonstrates in the creation of the NRC VAD Lexicon, this "comparative extreme selection" significantly reduces cognitive load. Humans are much better at identifying the outliers in a small group than they are at placing a single item on an infinite spectrum.
By aggregating these simple "best" and "worst" choices across many overlapping subsets, researchers can mathematically derive a highly reliable, continuous ranking for every item in the dataset.
Quantifying the Unquantifiable
This mechanism has allowed the field to move beyond binary classifications into "shades of gray" across several domains:
- Semantic Relatedness: In SemRel2024, Mohamed Abdalla and Nedjma Ousidhoum used BWS to measure how related text pairs are across 14 languages. This captures nuances like topic and entailment that simple similarity scores miss.
- Social Dynamics: David Jurgens and Jiaxin Pei applied BWS to Quantifying Intimacy in Language, turning a vague social dimension into a measurable metric.
- Bias and Stereotypes: Rather than just labeling text as "biased," Rishav Hada and colleagues used BWS to produce "Fifty Shades of Bias", a dataset of fine-grained gender bias scores. Similarly, Yang Liu used relative rankings to Quantify Stereotypes in Language on a continuous scale from -1 to 1.
Trade-offs in Reliability
While BWS produces superior gold standards, it is not a silver bullet. Yi Chu and Cassandra L. Jacobs found that BWS can be up to 4.5 times slower than categorical labeling. In their study on Detecting Implicit Bias, they noted that while BWS provides high-quality intensity scores, the sheer volume of comparisons required can be a bottleneck for large-scale datasets.
However, for tasks where quality and nuance are paramount—such as evaluating the Factual Consistency of Summaries or the Artistry of Literary Translation—the comparative framework remains the gold standard. As Xiangru Tang and Alexander Fabbri argue, when comparing high-quality models that are difficult to distinguish, forcing a discriminative choice through BWS is often the only way to reveal which system truly performs better.