Multi-Dimensional Quality Assessment for Generated Text

263 concepts210 papers1 proposal

When we evaluate a generated text, compressing its quality into a single scalar score is a lossy operation. A summary might be grammatically flawless yet entirely hallucinated. A creative essay might be highly original yet completely nonsensical. Traditional metrics like BLEU and ROUGE fail because they treat text quality as a monolithic, one-dimensional string-matching problem. As Jekaterina Novikova et al. argued in Why We Need New Evaluation Metrics for NLG, we need granular mappings of linguistic performance to understand where and why a model succeeds or fails.

This realization has driven the field toward multi-dimensional quality assessment: decomposing text evaluation into a vector of distinct, decoupled criteria. By evaluating independent axes such as fluency, coherence, factual consistency, and relevance, researchers can build metrics that align far more closely with human judgment.

Why Monolithic Metrics Fail

To understand why multi-dimensional assessment works, we must first look at why holistic or overlap-based metrics fail. In text simplification, Elior Sulem et al. demonstrated in BLEU is Not Suitable for the Evaluation of Text Simplification that standard machine translation metrics fail to capture whether a model actually simplifies sentence structure or merely preserves meaning. If an evaluation metric cannot separate grammaticality from structural simplicity, it rewards models that change nothing.

Similarly, in creative domains, Najoung Kim showed in Death of the Novel(ty): Beyond N-Gram Novelty as a Metric for Textual Creativity that relying on statistical n-gram novelty alone rewards nonsensical "slop" rather than meaningful creativity. Text quality is inherently multi-dimensional; evaluating it requires a decoupled approach that isolates specific linguistic properties.

The Core Mechanism: Decomposing the Quality Vector

Multi-dimensional assessment works by breaking down an abstract judgment into orthogonal axes. Depending on the domain, these axes target specific functional and stylistic requirements:

From Human Rubrics to LLM Judges

Executing multi-dimensional evaluation requires moving away from simple string overlap toward structured, aspect-aware scoring mechanisms. This is achieved through three primary methodologies:

1. Standardizing Human Judgment

Human evaluation is notoriously noisy. Pavlos Vougiouklis et al. found in How Biased Is Your NLG Evaluation? that crowdworkers systematically conflate fluency and coverage, leading to skewed results compared to experts. To fix this, Lining Zhang et al. demonstrated in A Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarization that explicitly training annotators on distinct dimensions like faithfulness, saliency, and coherence dramatically reduces variance and aligns crowd judgments with expert standards.

2. Prompt-Based LLM Evaluators

Rather than relying on human effort, modern frameworks use Large Language Models as multi-faceted judges. Sameer Jain introduced the ICE framework in Multi-Dimensional Evaluation of Text Summarization with In-Context Learning, showing that providing dimension-specific examples in an LLM's context window allows a single model to act as an accurate judge for consistency, relevance, and coherence independently. Similarly, Amila Ferron showed in MEEP: Is this Engaging? Prompting Large Language Models for Dialogue Evaluation in Multilingual Settings that embedding explicit, multi-dimensional definitions directly into prompts forces the model to decompose abstract concepts like "engagement" into measurable linguistic traits.

3. Structural and Unsupervised Probing

Multi-dimensional evaluation can also be framed as an unsupervised structural probe. In CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation, Hao Zhou et al. measured coherence and attribute relevance by decomposing them into targeted fill-in-the-blank tasks. Coherence was evaluated by masking a sentence and predicting it from its neighbors, while relevance was measured by the probability of generating a specific attribute label.

Why Decomposition Works: Eliminating the Halo Effect

The core intuition behind why multi-dimensional evaluation works so well is the elimination of the halo effect—a cognitive bias where an evaluator's impression of one quality influences their judgment of other, unrelated qualities. In text evaluation, an exceptionally fluent sentence often tricks a human or a naive model into thinking it is also factually accurate or relevant.

By forcing the evaluation framework to score dimensions independently, we break this correlation. For example, in specialized domains like medicine, David Fraile Navarro et al. showed in Expert evaluation of large language models for clinical dialogue summarization that automated neural evaluators assessing coherence, consistency, fluency, and relevance independently correlate much more closely with expert human clinicians than traditional metrics. By breaking down the evaluation, the system can reward a model for capturing critical clinical facts even if its phrasing differs from a reference note.

Furthermore, decomposition allows for multi-perspective simulations. Ning Wu et al. developed the DRPE framework in Large Language Models are Diverse Role-Players for Summarization Evaluation, which assigns different personas (e.g., professional critics vs. casual readers) to evaluate different dimensions of a text. This captures nuances like reader interest and factual consistency simultaneously, mirroring the multi-faceted way humans naturally consume and judge language.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_multi_dimensional_quality_assessment_08afc478,
  title = {multi-dimensional quality assessment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-dimensional-quality-assessment/rdr_08afc47898e94a83a548b981e7404744},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_multi_dimensional_quality_assessment_08afc478_v1,
  title = {multi-dimensional quality assessment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-dimensional-quality-assessment/rdr_08afc47898e94a83a548b981e7404744/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake