Abstract
Statistical methods applied to social media posts shed light on the dynamics of online dialogue. For example, users' wording choices predict their persuasiveness and users adopt the language patterns of other dialogue participants. In this paper, we estimate the causal effect of reply tones in debates on linguistic and sentiment changes in subsequent responses. The challenge for this estimation is that a reply's tone and subsequent responses are confounded by the users' ideologies on the debate topic and their emotions. To overcome this challenge, we learn representations of ideology using generative models of text. We study debates from 4Forums.com and compare annotated tones of replying such as emotional versus factual, or reasonable versus attacking. We show that our latent confounder representation reduces bias in ATE estimation. Our results suggest that factual and asserting tones affect dialogue and provide a methodology for estimating causal effects from text.
Introduction
In online discussions, we often observe "linguistic accommodation"—the tendency of participants to mirror each other’s style and sentiment. However, a major question remains: does a specific tone cause a change in the other person, or are both users simply reacting to their deep-seated ideological differences?
For example, if User A posts about gun control and User B replies "nastily," User A’s next post might become more negative. Is this because of User B’s nastiness (the treatment), or because their opposing ideologies (the confounder) naturally trigger negativity in both? Standard statistical correlations cannot distinguish between the two. This paper seeks to isolate the causal effect of tone by adjusting for these hidden "latent" confounders like ideology and intent.
Methodology
The authors frame the problem using the Potential Outcomes framework, analyzing "post triples": an initial post by User A, a reply by User B (the treatment), and a follow-up post by User A.
Latent Confounder Representation
Since "ideology" is not a single binary variable, the authors use Latent Dirichlet Allocation (LDA) to learn 50 latent topics from the debate text. These topics act as proxies for the specific "frames" or viewpoints that influence both the reply tone and the subsequent response. For instance, in a gun control debate, one user might focus on "constitutional rights" while another focuses on "gun violence."
Outcome Measurement and Robust Estimation
To represent posts, the authors use the Linguistic Inquiry and Word Count (LIWC) tool, which categorizes text into psychological and linguistic vectors (e.g., positive emotion, negations, first-person pronouns). The "outcome" is measured as the Euclidean distance between User A’s first and second post vectors.
To ensure findings are robust, three different estimators were compared:
- Standard Regression (MLE)
- Inverse Propensity Weighting (IPW): Reweighting samples based on the probability of a specific tone occurring.
- Augmented IPW (AIPW): A "doubly robust" method that yields consistent results if either the outcome model or the propensity model is correctly specified.
Experimental Setup
The study utilizes the 4FORUMS.COM dataset, part of the Internet Argument Corpus, covering polarizing topics such as abortion, gay marriage, evolution, and gun control. The dataset includes human-annotated labels for reply types, including nasty vs. nice, attacking vs. reasonable, emotional vs. factual, and questioning vs. asserting. Approximately 1,500 post triples were utilized for the final analysis.
Key Results
The Stabilizing Effect of Factual Discourse
The most significant finding is that factual and assertive tones have a clear causal effect: they lead to smaller changes in the original poster's linguistic style and sentiment. Factual replies seem to keep the conversation "on track." When a user receives a factual response, they are less likely to shift their emotional state or linguistic style, suggesting that factual discourse may act as a stabilizing force in polarized environments.
As shown in the results table, factual replies caused a significant decrease in linguistic style changes (AIPW estimate of -1.2) and a decrease in negative sentiment shifts. Assertive tones similarly reduced the variance in the follow-up post's style (AIPW estimate of -1.2).
Nastiness as a Confounded Variable
Interestingly, while it is often assumed that "nastiness" drives online vitriol, the causal evidence for it was weak once ideology was controlled for. After adjusting for latent ideology, the causal effect of a "nasty" tone on a user's sentiment change was not statistically significant. This suggests that the observed correlation between nastiness and negative responses in previous studies might be driven more by underlying disagreement than the tone itself.
Discussion and Limitations
While the latent confounder representation (Z-Full) significantly reduces bias compared to naive approaches, the results rely on the "ignorability" assumption—the hope that the LDA model captured all relevant confounders. If other hidden factors, such as the users' prior relationship, are not captured in the text, the estimates could still be biased.
Future work may explore deep generative models to learn even more complex confounder representations and investigate the "position" of a post within a long thread as an additional factor in dialogue dynamics.
Summary updated