An Empirical Study of the Influence of Argument Conciseness on Argument Effectiveness

Giuseppe Carenini, Johanna D. MooreACL 2000 · DOI

Abstract

Determining the optimal level of detail is a fundamental challenge in Natural Language Generation (NLG), particularly for systems designed to influence user opinions. This study investigates the relationship between argument conciseness and effectiveness through the lens of decision theory. By utilizing the IDEA system to generate tailored evaluative arguments, the researchers demonstrate that "less is more": concise arguments significantly outperform verbose ones in shifting user attitudes. The findings suggest that excessive evidence leads to a "dilution effect," where weaker supporting points distract from more compelling information.

1. Introduction

While NLG systems have become proficient at generating fluent text, measuring the persuasive effectiveness of that text remains difficult. Argumentation theory posits that arguments should be concise, yet provides little quantitative guidance on the ideal amount of detail. Providing too little information may fail to convince, while providing too much may overwhelm or bore the user. This paper seeks to empirically identify the impact of conciseness on user satisfaction with recommended options in the real-estate domain.

2. Methodology

The researchers developed a comprehensive pipeline to generate and evaluate arguments based on individual user preferences.

2.1 User Modeling and Compellingness

The system employs an Additive Multiattribute Value Function (AMVF) from decision theory to model user preferences across various house attributes (e.g., commute length, garden size). To select which evidence to include, the authors defined a metric called s-compellingness, which calculates the strength of a specific piece of evidence relative to the user's modeled priorities.

2.2 The Conciseness Parameter ($k$)

A central innovation of this work is the introduction of a conciseness threshold, $k$. By adjusting $k$, the system can control the number of evidence points included in the final text. A high $k$ value results in a highly concise argument focusing only on the most compelling points, while a lower $k$ value generates a more verbose argument.

Examples of generated arguments at varying levels of conciseness (k)

3. Experimental Design

The study utilized the IDEA system, which monitors user interactions with a data visualization tool to refine the underlying user model. A between-subjects experiment was conducted with 30 participants across three conditions:

  • No-Argument: Users were presented with raw data only.
  • Tailored-Concise: Arguments generated with $k=-0.3$ (averaging 10 pieces of evidence).
  • Tailored-Verbose: Arguments generated with $k=-1$ (averaging 16 pieces of evidence).

4. Key Results and Analysis

The primary metric for effectiveness was the satisfaction z-score, measuring user preference for the recommended house relative to other top options. The results indicate that concise arguments are significantly more effective than verbose ones ($p=0.02$) and the no-argument baseline ($p=0.03$).

  • Tailored-Concise: 0.88
  • No-Argument: 0.25
  • Tailored-Verbose: 0.05
Relative effectiveness of the three experimental conditions

4.1 The Dilution Effect

The data supports the existence of a "dilution effect" in automated argumentation. In the verbose condition, the inclusion of additional, weaker evidence appeared to distract users from the most compelling points, making the overall argument less persuasive than if those points had been omitted. Notably, verbose arguments performed worse than providing no argument at all, suggesting that poor argumentation can be counterproductive.

5. Conclusion

This research provides empirical evidence that conciseness is a critical factor in the effectiveness of generated evaluative arguments. By quantifying subjectivity through Multiattribute Utility Theory (MAUT), the authors successfully created a tunable system for persuasion. Future work should explore the generalizability of these findings across different domains and larger participant cohorts to pinpoint the absolute optimal level of detail for various decision-making contexts.

Summary updated

BibTeX
@inproceedings{carenini2000empirical,
  author = {Giuseppe Carenini and Johanna D. Moore},
  title = {{An Empirical Study of the Influence of Argument Conciseness on Argument Effectiveness}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2000},
  doi = {10.3115/1075218.1075238},
}
Figure