Evaluating Subjective Qualities in Dialogue and Argumentation

92 concepts68 papers

In goal-oriented natural language processing, success is often binary and easily verifiable. For instance, in a flight-booking system like the one studied by Li in AirDialogue: An Environment for Goal-Oriented Dialogue Research, an agent either satisfies the user's constraints or it does not. There is a mathematically defined ground truth.

However, when we transition to open discourse—such as open-domain dialogue, role-playing, social commentary, or computational argumentation—this ground truth vanishes. How do you objectively measure if a chatbot's response is "interesting" or if a legal argument is "convincing"? As Ashley et al. noted in their early work on adversarial legal reasoning (Compare and Contrast: A Test of Expertise), determining the relative quality of competing arguments is inherently subjective. This subjectivity creates a fundamental bottleneck: if human experts cannot agree on what makes a response or argument "good," developing automated evaluation functions becomes exceptionally difficult.

The Dimensions of Disagreement

To understand why evaluating open discourse is so challenging, we must look at where human judgments diverge. Subjectivity is not just random noise; it is driven by several systematic factors:

Why Traditional Metrics Fail

When evaluating abstract qualities like "engagingness," "personality consistency," or "constructiveness," traditional automated metrics break down. As highlighted by Kumar et al. in AMUSED: A Multi-Stream Vector Representation Method for Use in Natural Dialogue, algorithmic benchmarks fail to capture whether a conversation is genuinely interesting, forcing researchers to rely on expensive and unscalable human evaluations.

Even when human evaluation is used, standard crowd-sourcing methodologies frequently suffer from a lack of calibration. See et al. (What makes a good conversation? How controllable attributes affect human judgments) observed high subjectivity and low calibration among human raters judging dialogue engagement. This issue is compounded when using absolute numeric scales (e.g., rating constructiveness from 1 to 10). As shown by Fujita et al. in Dataset Creation for Ranking Constructive News Comments, absolute rating scales vary so significantly between individual annotators that they fail to provide a reliable ground truth for training machine learning models.

Furthermore, open-domain conversations lack a unique structural mapping. The authors of Structured Attention for Unsupervised Dialogue Structure Induction note that real-world human conversations can be mapped in multiple valid ways, making it exceptionally difficult to establish standardized performance metrics for dialogue structure induction. This ambiguity extends to downstream tasks like summarization; Chen et al. (DialogSum Challenge: Summarizing Real-Life Scenario Dialogues) note that summarizing dialogues objectively becomes difficult when the conversation contains inherently subjective or unverified statements.

Algorithmic Solutions for Subjective Spaces

To build reliable models in the face of this inherent subjectivity, researchers have shifted from trying to eliminate disagreement to modeling it. Several key mechanisms have emerged to handle this transition.

Unanimity-Aware and Consensus Metrics

Instead of treating low inter-annotator agreement as a failure, some frameworks use it as a core signal. In the Overview of the NTCIR-13 Short Text Conversation Task, Shang et al. introduced evaluation frameworks that incorporate unanimity-aware metrics. By weighting evaluation scores based on the level of agreement among multiple human judges, the metric prioritizes responses that are universally recognized as high-quality. This strategy shifts the goal from finding a single "correct" answer to identifying universally acceptable responses.

Pairwise and Relative Ranking

Because humans are notoriously poor at assigning absolute scores but excellent at making relative comparisons, shifting the task framing can drastically improve reliability. Toledo et al. explored this in Automatic Argument Quality Assessment - New Datasets and Methods, examining whether argument quality is better captured through absolute individual scores or relative pairwise comparisons. Pairwise ranking mitigates individual annotator calibration bias, transforming a vague subjective judgment into a concrete choice ("Is argument A more convincing than argument B?"), which allows models to learn a more stable relative ordering of argumentative strength.

Multidimensional Taxonomies and Persona Constraints

When evaluating complex conversational agents, such as role-playing models, success cannot be measured by factual correctness. Tu et al. demonstrate in CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation that role-playing success must be evaluated through subjective traits like emotional resonance and stylistic consistency.

To handle this, researchers break down abstract concepts into explicit, multidimensional taxonomies. For instance, Ferron et al. introduce a structured taxonomy of "engagingness" in MEEP: Is this Engaging? Prompting Large Language Models for Dialogue Evaluation in Multilingual Settings, allowing large language models to evaluate dialogue quality more precisely across different languages by focusing on specific behavioral dimensions rather than a single vague impression.

Human Preference Rewards

Finally, instead of optimizing for token-matching metrics like BLEU, modern architectures optimize directly for human preference distributions. In ViCo: Engaging Video Comment Generation with Human Preference Rewards, Sun et al. utilize a reward model trained on human engagement signals (such as "likes") to guide reinforcement learning. This allows the system to generate subjective, engaging social commentary without relying on a single, rigid ground-truth text, effectively letting the model learn the implicit distribution of human preference.

Key Foundational Work

Compare and Contrast: A Test of Expertise (1987) — contributed 2 concept elements to this direction.
A Grammatico-Statistical Approach to Discourse Partitioning (1994) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_subjectivity_in_rhetorical_assessment_f367c428,
  title = {subjectivity in rhetorical assessment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/subjectivity-in-rhetorical-assessment/rdr_f367c428db1c44adba6e01a060e26afa},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_subjectivity_in_rhetorical_assessment_f367c428_v1,
  title = {subjectivity in rhetorical assessment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/subjectivity-in-rhetorical-assessment/rdr_f367c428db1c44adba6e01a060e26afa/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake