On the quality of persuasion dialogs

Leïla Amgoud, Florence Dupin de Saint-Cyr2011

TL;DR

This paper introduces a framework for objectively evaluating the quality of persuasion dialogues. It proposes a set of measures to assess argument quality, agent behavior (coherence, aggressiveness), and overall dialog properties like relevance and conciseness, moving beyond mere protocol adherence to assess the effectiveness of persuasive exchanges.

What Problem Does This Paper Address?

Current research on persuasion dialogues primarily focuses on the protocols and strategies that govern how agents interact. While these protocols ensure dialogues are well-formed and terminate, they offer little insight into the quality of the dialogue itself. Two agents following the same protocol can produce vastly different dialogues, some more persuasive or informative than others. This paper addresses the critical gap of objectively evaluating and comparing persuasion dialogues, aiming to provide criteria that can guide the generation of better dialogues and analyze existing ones. The challenge lies in defining quantifiable metrics for subjective qualities like "persuasiveness" or "effectiveness."

Methods

The authors propose a multi-faceted approach to evaluating persuasion dialogues, categorizing their measures into three main families:

  1. Quality of Exchanged Arguments: This involves assigning numerical weights to arguments, reflecting their quality (e.g., certainty, cost of information). The overall dialog weight is the sum of its arguments' weights.
  2. Behavior of Each Participating Agent:
    • Aggressiveness: Measured by the proportion of an agent's arguments that attack the other agent's arguments.
  3. Loan (Borrowing): Quantifies how much an agent relies on formulas introduced by other agents for their own arguments, indicating strategic behavior.
  4. Coherence: Assesses self-contradiction, both explicit (presenting conflicting arguments) and implicit (through the complete argumentation system derived from the agent's utterances).
  5. Quality of the Dialog Itself:
    • Relevance and Usefulness: Moves are deemed relevant if their argument is linked to the dialog's subject, and useful if there's a directed path from the argument to the subject in the argumentation graph.
  6. Decisiveness: A move is decisive if its removal changes the dialog's outcome (whether the subject is accepted or rejected).
  7. Conciseness: The paper introduces the concept of an "ideal dialog," a concise sub-dialog that captures the essential elements for determining the subject's status. The closer a dialog is to its ideal version, the better it is. This is achieved by constructing dialog trees and identifying canonical trees that represent minimal sets of arguments sufficient to determine the outcome.

Datasets

Not discussed in the paper. The paper focuses on theoretical frameworks and evaluation metrics, using illustrative examples rather than empirical datasets.

Key Insights and Ideas

  • Objective Evaluation is Possible: The paper demonstrates that subjective qualities of persuasion dialogues can be objectively measured using formal argumentation theory.
  • Beyond Protocol Compliance: Dialogue quality is distinct from protocol adherence. A well-formed dialogue isn't necessarily a good one.
  • Multi-Dimensional Quality: Dialogue quality is not a single metric but a combination of argument strength, agent behavior, and dialog structure.
  • "Ideal Dialog" as a Benchmark: The concept of an "ideal dialog" provides a target for conciseness and effectiveness, serving as a benchmark for evaluating real dialogues.
  • Decisive Moves are Key: Identifying "decisive" moves highlights arguments that are crucial for the dialogue's outcome, suggesting that dialogues should prioritize these.
  • Conciseness Enhances Efficiency: Ideal dialogues, being concise, are more efficient as they contain only useful and relevant arguments.

Key Results

The paper defines several measures and properties:

  • Argument Weighting: A numerical weight can be assigned to arguments, allowing for the calculation of dialog weight and agent contribution.
  • Aggressiveness Measure: A formal definition for an agent's aggressiveness towards another agent is provided.
  • Loan Degree: A metric to quantify how much an agent borrows from others, offering insights into strategic behavior.
  • Incoherence Measure: A formal definition for an agent's incoherence degree, capturing both explicit and implicit contradictions.
  • Relevance, Usefulness, and Decisiveness: Criteria for evaluating individual moves within a dialogue.
  • Dialog Trees and Canonical Trees: A method to extract minimal, relevant sub-dialogues (canonical trees) that preserve the original dialog's outcome.
  • Ideal Dialog: Defined as the smallest canonical tree that determines the subject's status, serving as a benchmark for conciseness and effectiveness.

Limitations and Future Work

  • Subjectivity of Weights: The paper acknowledges that argument weights are assigned by an external agent and can be subjective, though it aims to provide a framework for their use.
  • Computational Complexity: While not explicitly detailed, the construction of dialog trees and canonical trees could be computationally intensive for very large dialogues.
  • Protocol Independence: The proposed measures are designed to be protocol-independent, but the paper notes that future work could explore how protocols can be designed to generate higher-quality dialogues based on these criteria.
  • Complete Argumentation Systems: The paper suggests exploring dialog trees based on "complete" argumentation systems, which might lead to more efficient dialogues but could require agent cooperation.
  • Order of Arguments: The current framework primarily focuses on the set of arguments exchanged, not their order. Future work could incorporate criteria related to the sequence of arguments for analyzing dialog quality.

Summary updated

Abstract

Several systems have been proposed for generating persuasion dialogs in which agents try to persuade each others to change their mind on a state of affairs. In this paper, we focus on the evaluation of the quality of those dialogs. We particularly propose three families of measures: i) measures of the quality of exchanged arguments, ii) measures of the behavior of each participating agent in terms of coherence, aggressiveness and the novelty of her arguments, iii) measures of the quality of the dialog itself in terms of relevance and usefulness of its moves. A notion of conciseness of a dialog is also introduced. For each persuasion dialog, we compute its ideal dialog which is a concise sub-dialog. The closer a dialog to its ideal sub-dialog, the better it is. 1
BibTeX
@misc{amgoud2011quality,
  author = {Leïla Amgoud and Florence Dupin de Saint-Cyr},
  title = {{On the quality of persuasion dialogs}},
  year = {2011},
}
Figure