When we measure complex, subjective qualities—such as the fluency of a language learner, the coherence of a chatbot, or the relevance of a research paper—we rely on rating scales. But how many points should that scale have? If a scale is too coarse (e.g., binary or a 3-point Likert scale), we lose the ability to distinguish subtle variations. If it is too granular (e.g., a 31-point scale), human judges become overwhelmed, and machine judges struggle to remain consistent.
This tension defines the study of rating scale granularity. Choosing the right resolution is not merely a matter of preference; it fundamentally dictates the quality, variance, and reliability of the resulting data.
The Failure of Coarse Scales: Ceiling Effects and Compression
Why can't we just use simple 3-point or 5-point scales for everything? The primary issue is the ceiling effect and score compression. When evaluating high-performing systems or advanced populations, scores naturally cluster at the top of a restricted range, making it impossible to separate the exceptional from the mediocre.
In oral proficiency testing, Febe de Wet et al. demonstrated this limitation in Automatic large-scale oral language proficiency assessment. They found that coarse human grading scales caused student scores to cluster heavily at the higher end. This lack of resolution obscured the true variance in student ability and weakened correlations with objective metrics like rate of speech.
A similar phenomenon occurs when crowdsourcing evaluations for generative AI models. As Xiangru Tang et al. observed in Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries, absolute Likert scales lack the granularity required to distinguish between high-quality text summaries that are closely matched. This forces annotators to assign identical top marks, leading to compressed score distributions and low inter-annotator agreement.
This compression problem spans multiple domains:
- Dialogue Systems: Ye Zheng et al. argued in Towards Quantifiable Dialogue Coherence Evaluation that restricted quality levels fail to capture the full nuance of human conversation, requiring finer-grained quantification to align models with human perception.
- Educational Diagnostics: In Item Response Ranking for Cognitive Diagnosis, Qi Liu et al. noted that insufficiently discriminated proficiency scores make students with distinct actual skill levels look identical, destroying the utility of personalized feedback.
- Phonetics: Even in perceptual speech science, Katrin Schneider et al. showed in Perceptual magnet effect in German boundary tones that large step sizes on an evaluation scale can completely obscure subtle linguistic phenomena, such as the perceptual magnet effect in boundary tones.
The Human-Machine Divergence in High Granularity
If coarse scales fail, the intuitive solution is to expand the scale. However, the success of a high-resolution scale depends heavily on whether the evaluator is a human or a Large Language Model (LLM).
For human evaluators, high granularity introduces a heavy cognitive tax. Christian Jensen et al. explicitly mapped this trade-off in Choosing a scale for measuring perceived prominence. When measuring perceived word prominence in speech, they discovered that a highly granular 31-point scale significantly increased listener effort, cognitive load, and task completion times for untrained listeners, without providing any proportional gain in data quality over a simple 4-point scale.
Yet, annotator expertise changes this dynamic. Denis Arnold et al. found in Obtaining prominence judgments from naive listeners.influence of rating scales linguistic levels and normalisation that 11-point and 31-point scales can yield richer, more robust insights and better acoustic correlations than binary scales, because they allow judges to separate their perception of a linguistic phenomenon from their internal confidence about the decision.
Low Granularity (e.g., 2-5 points)
[ Low Cognitive Load ] --------> [ High Score Compression / Ceiling Effects ]
High Granularity (e.g., 11-31 points)
[ High Cognitive Load ] -------> [ Richer Nuance (Experts) OR Inconsistency (Naive) ]
For LLMs, the trade-offs shift. When tasks require complex reasoning over multi-faceted criteria, high-resolution scales give models the "breathing room" to express subtle differences. For instance, Christian Jaumann et al. utilized an expanded 0–19 Likert scale for zero-shot abstract screening in LGAR: Zero-Shot LLM-Guided Neural Ranking for Abstract Screening in Systematic Literature Reviews. The authors demonstrated that this high-resolution scale allowed the LLM to express nuanced differences in document relevance that low-resolution scales missed entirely.
However, LLMs still hit a resolution ceiling when forced to mimic human grading structures. In Rating Short L2 Essays on the CEFR Scale with GPT-4, Kevin P. Yancey et al. observed that while GPT-4 could utilize the full range of the CEFR language proficiency scale, it consistently struggled to differentiate between adjacent levels (such as borderline B2 vs. true B2) compared to human experts. This mirrors a persistent challenge in automated prosody evaluation noted by Heini Kallio et al. in Prominence-based Evaluation of L2 Prosody, where models fail to reliably distinguish adjacent proficiency levels at the high end of the spectrum.
Bypassing the Trade-off via Alternative Architectures
To get the resolution of a fine-grained scale without the cognitive load or model instability of a holistic slider, researchers have designed alternative evaluation architectures.
1. Relative Ranking over Absolute Rating
Instead of asking an evaluator to place an item on an absolute scale, frameworks can force a relative comparison. Xiangru Tang et al. showed that Best-Worst Scaling (a pairwise ranking protocol) is far more reliable than Likert scales for distinguishing high-quality, closely matched model outputs. Similarly, Qi Liu et al. bypassed flat classification by introducing an Item Response Ranking framework, optimizing for monotonicity to ensure that higher proficiency naturally maps to a higher relative ranking, drastically improving score discrimination.
2. Compositional Granularity (Binary Batteries)
An elegant solution to the granularity dilemma is to break a single complex, high-resolution judgment down into a battery of simple, low-resolution pieces.
Holistic High-Resolution Scale (Hard for humans/models to calibrate consistently):
Score: [ 1 . 2 . 3 . 4 . 5 . 6 . 7 . 8 . 9 . 10 ]
Compositional Granularity (Easy to calibrate, high-resolution output):
[Question 1: Yes/No] + [Question 2: Yes/No] + [Question 3: Yes/No] + ... = Fine-Grained Score
This is the core mechanism behind the BINEVAL framework introduced by Sangwoo Cho et al. in Ask, Don’t Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement. Instead of asking an LLM judge to rate an output on an opaque, holistic 1-to-5 scale, BINEVAL poses a battery of independent, atomic binary ("yes/no") questions.
When these binary checks are aggregated, they naturally produce a fine-grained, high-variance score distribution that closely mirrors human variance. This effectively mitigates the ceiling effect while keeping individual evaluation steps trivial—proving that achieving high resolution does not require a high-resolution scale.
Key Foundational Work
Choosing a scale for measuring perceived prominence (2005) — contributed 7 concept elements to this direction.
Automatic large-scale oral language proficiency assessment (2007) — contributed 2 concept elements to this direction.
Perceptual magnet effect in German boundary tones (2005) — contributed 1 concept elements to this direction.