StoryARG: a corpus of narratives and personal experiences in argumentative texts

Neele Falk, Gabriella LapesaACL 2023 · DOI

Abstract

Researchers have introduced StoryARG, a novel dataset of over 2,400 text spans that explores how narratives and personal experiences function within arguments. Their analysis reveals that while storytelling is a powerful tool for persuasion—especially when proposing solutions—the perceived "effectiveness" of a story is deeply subjective and varies significantly between individual listeners.

Introduction

In the field of computational argumentation, researchers typically focus on "rational" elements: logic, verifiable facts, and formal premises. However, in real-world "everyday" arguments—from Reddit debates to policy discussions—humans constantly use stories and personal anecdotes to make their points.

Until now, these narratives were often dismissed as "second-class" evidence because they aren't strictly logical or easily verifiable. This paper addresses the lack of high-quality, fine-grained data on how stories actually work in an argument. Do they make an argument more persuasive? What specific narrative properties (like having a clear plot or a specific protagonist) contribute to an argument's success?

The StoryARG Corpus

StoryARG is a "corpus of corpora," compiling 2,451 annotated spans from 507 documents across four diverse sources:

  • ChangeMyView (CMV): High-intensity Reddit debates.
  • RegulationRoom: Online public comments on US government regulations regarding airline peanut policies and debt collection practices (cdcp).
  • Europolis: Transcripts of face-to-face deliberative discussions between European citizens.
  • NYT Comments: User comments on New York Times articles regarding veganism.
Table 1: Overview of the StoryARG corpus sources, genres, and document counts.

Methodology

The authors developed an interdisciplinary annotation schema bridging linguistics, social science, and narrative theory. Text spans were annotated at two distinct levels:

  1. Argumentative Level: Focuses on the story's purpose (e.g., clarifying a point, disclosing harm, searching for a solution, or establishing authority). It also includes ratings for emotional load and effectiveness on a 3-point scale.
  2. Narrative Level: Examines structural properties such as plot presence, protagonist type (individual vs. group), factuality (factual vs. hypothetical), and perspective (first-hand vs. second-hand).

To understand what drives effectiveness, the authors performed a regression analysis, using these annotated properties as independent variables to predict the "effectiveness" scores given by human annotators.

Results: Determinants of Persuasive Effectiveness

The analysis yielded several key insights into how narratives influence argumentative success:

  • Functional Impact: Stories illustrating a solution to a problem are perceived as significantly more effective than those that merely clarify a position or establish background.
  • Structural Importance: "Stories" (defined as having a clear plot-like development) are rated as more effective than "Experiential Knowledge" (fragmented mentions of personal experience).
  • The Credibility Gap: While fictional or hypothetical stories can clarify a point, they are less effective for establishing authority or disclosing harm, where audiences demand factual authenticity.
Figure 1: Regression estimates for features predicting the effectiveness of narrative spans in arguments.

Quantitatively, the final model explained 31% of the variance in effectiveness ratings. Notably, the single strongest predictor was the identity of the annotator (13.41%), followed by the argumentative function (4.38%) and narrative type (3.42%). This highlights that persuasion is inherently subjective; what moves one person might leave another cold.

Discussion and Future Work

The authors acknowledge that the dataset is relatively small for training large-scale machine learning models. Furthermore, the inter-rater agreement for "effectiveness" and "argumentative function" was low, reflecting the inherent subjectivity of the task.

Future work could involve larger-scale crowdsourcing to better capture "perspectivism"—how different demographic groups react to the same stories. The authors also suggest that their dataset can now be used to train models that automatically detect and classify the functional roles of stories in digital text, potentially aiding in the moderation of online deliberative platforms.

Summary updated

BibTeX
@inproceedings{falk2023storyarg,
  author = {Neele Falk and Gabriella Lapesa},
  title = {{StoryARG: a corpus of narratives and personal experiences in argumentative texts}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.132},
}
Figure