Having the Last Word: Understanding How to Sample Discussions Online

Gioia Boschi, Anthony P. Young, Sagar Joglekar, Chiara Cammarota, Nishanth SastryarXiv 2019 · arXiv

Abstract

This paper applies formal argumentation theory and complex network modeling to determine which comments in massive online discussions are "normatively justified" (i.e., successfully defended or unrebutted). By modeling discussions as trees, the researchers identify specific patterns—based on the "acrimony" of the debate—that tell readers exactly where in a thread they are most likely to find the most valid arguments.

1. Introduction

Online discussions on platforms like Reddit, Twitter, or Kialo often scale to thousands of comments, creating a massive "information overload." A typical user can only read a small sample of the thread. However, if a user samples randomly or chronologically, they might miss crucial rebuttals, leading them to believe an argument is valid when it has actually been effectively debunked elsewhere in the tree.

The authors tackle the challenge of principled sampling: How can we identify the "justified" positions in a sprawling debate without reading every single post? They seek to understand how the structure of a discussion (its "tree" shape) and the nature of its interactions (support vs. attack) influence the location of these justified arguments.

2. Methodology and Formal Framework

The researchers utilize Bipolar Argumentation Frameworks (BAFs) to model the complexities of online debate. In this model, a discussion is a directed graph where nodes are arguments and edges represent either "support" or "attack."

Example of a Bipolar Argumentation Framework (BAF) showing support (green dotted) and attack (red solid) relations between arguments.

To determine the validity of arguments, the authors employ a two-step process:

  1. Conversion: They convert these BAFs into Dung’s standard Argumentation Framework (AF). This process translates support relations into additional attack chains (e.g., if A supports B and B attacks C, then A is considered to attack C).
  2. Labeling: They identify "justified" arguments—those that are either unrebutted or defended by other justified arguments—using a polynomial-time algorithm that propagates labels from the leaves (unrebutted points) back up to the root.
The corresponding Argumentation Framework (AF) derived from the BAF, where support relations are converted into direct or mediated attacks.

3. Modeling Discussion Dynamics

The study analyzes two types of discussion trees: homogeneous trees (consistent branching) and non-homogeneous (scale-free) trees generated via preferential attachment to mimic real-world social dynamics. They introduce a parameter $q$, representing the probability that a reply is supportive, to derive the probability $p_h$ that a comment at a certain depth (level $h$) is justified.

4. Empirical Analysis: The Kialo Dataset

The study uses a real-world dataset scraped from Kialo, an online debating platform. Unlike Twitter or Reddit, Kialo is highly structured; users must explicitly label their replies as "pro" (support) or "con" (attack). The authors extracted 1,511 final discussion trees, with an average of 204 arguments per thread and relatively balanced debate dynamics ($q$ between 0.4 and 0.6).

5. Key Insights and Results

  • The "Last Word" Effect: In argumentation theory, the "leaves" of the tree (comments with no replies) are justified by default. Because these leaves are so numerous in scale-free networks, they exert an enormous influence on which arguments deeper in the tree are considered justified.
  • Supportive Discussions ($q > 0.5$): The model predicts that justified arguments are clustered at the "bottom" of the tree. For readers, this suggests that reverse-chronological sorting is a decent heuristic for finding justified points in supportive threads.
  • Acrimonious Discussions ($q < 0.5$): In "aggressive" discussions where attacks outnumber supports, the probability of an argument being justified oscillates based on its distance from the leaves. Justified arguments are scattered throughout, meaning there is no single "best" level to sample.
  • Structural Dominance: In real-world trees, the actual distribution of justified arguments is dominated more by the location of the "leaves" than by the specific ratio of support vs. attack.

6. Limitations and Future Work

A primary limitation is the "vacuously justified" problem: the model assumes any unrebutted comment is "justified," ignoring the possibility that a comment was simply too low-quality to warrant a reply. Future research aims to incorporate preference-based frameworks, where "likes" or user reputation influence argument weight, and to design UI elements that highlight "weak" arguments that are currently justified but vulnerable to a single new attack.

Summary updated

Abstract

In online debates individual arguments support or attack each other, leading to some subset of arguments being considered more relevant than others. However, in large discussions readers are often forced to sample a subset of the arguments being put forth. Since such sampling is rarely done in a principled manner, users may not read all the relevant arguments to get a full picture of the debate. This paper is interested in answering the question of how users should sample online conversations to selectively favour the currently justified or accepted positions in the debate. We apply techniques from argumentation theory and complex networks to build a model that predicts the probabilities of the normatively justified arguments given their location in online discussions. Our model shows that the proportion of replies that are supportive, the number of replies that comments receive, and the locations of un-replied comments all determine the probability that a comment is a justified argument. We show that when the degree distribution of the number of replies is homogeneous along the discussion, for acrimonious discussions, the distribution of justified arguments depends on the parity of the graph level. In supportive discussions the probability of having justified comments increases as one moves away from the root. For discussion trees that have a non-homogeneous in-degree distribution, for supportive discussions we observe the same behaviour as before, while for acrimonious discussions we cannot observe the same parity-based distribution. This is verified with data obtained from the online debating platform Kialo. By predicting the locations of the justified arguments in reply trees, we can suggest which arguments readers should sample to grasp the currently accepted opinions in such discussions. Our models have important implications for the design of future online debating platforms.
BibTeX
@misc{boschi2019having,
  author = {Gioia Boschi and Anthony P. Young and Sagar Joglekar and Chiara Cammarota and Nishanth Sastry},
  title = {{Having the Last Word: Understanding How to Sample Discussions Online}},
  year = {2019},
  eprint = {1906.04148},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.04148},
}
Figure