Affective Behaviour Analysis of On-line User Interactions: Are On-line Support Groups More Therapeutic than Twitter?

Giuliano Tortoreto, Evgeny Stepanov, Alessandra Cervone, Mateusz Dubiel, Giuseppe RiccardiACL 2019

TL;DR

This paper introduces a novel NLP-based methodology to automatically identify therapeutic factors in online conversations, specifically comparing online support groups (OSGs) with Twitter. The study finds that OSGs exhibit a significantly higher presence of these therapeutic factors, suggesting they may be more beneficial for users seeking psychological support.

What Problem Does This Paper Address?

The increasing prevalence of mental health issues has led many individuals to seek support online. While online support groups (OSGs) and social media platforms like Twitter offer avenues for this, their therapeutic potential and the mechanisms behind any benefits are not fully understood. Previous research often relied on manual annotation, which is time-consuming and limits the scale of analysis. This paper addresses the need for an automated method to quantify therapeutic factors in online interactions, enabling a broader comparison between different platforms and a deeper understanding of their impact on users' emotional well-being.

Methods

The authors propose a methodology that leverages Natural Language Processing (NLP) techniques, specifically Sentiment Analysis and Dialogue Act tagging, to automatically identify three key therapeutic factors identified by Yalom and Leszcz (2005): Universality, Altruism, and Instillation of Hope.

  • Universality: Identified by posts that disconfirm a user's feeling of uniqueness in their condition, often through shared negative experiences.
  • Altruism: Detected when users offer support, reassurance, or insight to others, especially in response to negative experiences.
  • Instillation of Hope: Inferred when users share positive personal experiences that inspire others and increase their expectation of positive outcomes.

The methodology involves:

  1. Pre-processing: Extracting conversation threads from asynchronous multi-party conversations.
  2. Dialogue Act Tagging: Analyzing the illocutionary force (speaker's intention) of utterances.
  3. Sentiment Analysis: Determining the emotional tone of the text.
  4. Quantitative Analysis: Applying these NLP techniques to identify conditions indicative of the therapeutic factors.

This automated approach allows for the analysis of a much larger volume of data compared to manual annotation.

Datasets

The study utilizes two datasets:

  • Twitter: A collection of 1,873 Twitter conversation threads, comprising approximately 14,000 tweets, previously pre-processed to extract conversation threads.
  • Online Support Groups (OSG): A dataset developed by crawling and pre-processing an OSG web forum. This dataset is significantly larger, containing 295,000 conversations with an average of 6 comments each, totaling 1.5 million comments.

The OSG dataset is noted for its multi-threaded nature, requiring a specific thread extraction algorithm to ensure a fair comparison with the Twitter dataset.

Key Insights and Ideas

  • OSGs show greater therapeutic potential: The core finding is that OSGs exhibit a higher frequency of therapeutic factors compared to Twitter conversations. This suggests that platforms specifically designed for support may be more conducive to positive emotional outcomes.
  • Automated analysis is feasible and scalable: The paper demonstrates that NLP techniques can effectively approximate the identification of complex therapeutic factors, overcoming the limitations of manual annotation.
  • Sentiment shift is a key indicator: A significant increase in the original poster's sentiment score at the end of a conversation thread is a strong indicator of therapeutic interaction. OSGs show a much higher rate of sentiment increase.
  • Negative experiences can be therapeutic: In OSGs, even negative or neutral comments can lead to positive sentiment shifts for the original poster, highlighting the value of shared experiences and empathy. This contrasts with Twitter, where only positive comments tend to lead to positive outcomes.
  • Dialogue Acts reveal interaction patterns: The analysis of Dialogue Acts reveals that users in OSGs are more likely to engage in supportive and empathic communication, as indicated by a higher frequency of "accept" and "reject" tags when discussing others' points of view.

Key Results

  • Sentiment Change: In OSGs, conversations leading to an increase in the original poster's sentiment score are considerably more frequent than those leading to a decrease. In contrast, Twitter shows a roughly equal distribution of sentiment increases and decreases.
  • Polarity Shift: In OSGs, users are more likely to change their polarity from negative to positive. In Twitter, users are more likely to change from positive to negative.
  • Dialogue Act Distribution: OSG conversations show a higher prevalence of "accept" and "reject" Dialogue Act tags compared to Twitter, suggesting more nuanced engagement with others' viewpoints.
  • Overall Therapeutic Factor Presence: The study concludes that OSG conversations satisfy a higher number of conditions approximating the selected therapeutic factors (Universality, Altruism, Instillation of Hope) than Twitter conversations.

Limitations and Future Work

  • Necessary but not Sufficient Conditions: The authors acknowledge that the identified conditions for therapeutic factors are necessary but not sufficient. The analysis focuses on the structure of conversations and is agnostic to the specific content, which can be a limitation.
  • Differentiating Therapeutic Factors: The current methodology cannot precisely differentiate between closely related therapeutic factors like Altruism and Instillation of Hope, as this requires distinguishing between a user's emotional state and their intended emotional communication.
  • Comparison to Human Evaluation: The proposed approach is an approximation of expert annotation. Future work should involve comparing the automated analysis results with human evaluations to validate its accuracy.
  • Augmentation with Other Metrics: The authors suggest augmenting the study with other language analysis metrics and correlating them with human annotations to strengthen the approximations.
  • Personalized Support: The methodology is envisioned as a tool to supplement manual analysis, providing general practitioners and psychologists with additional information to offer more personalized support to individuals.

Summary updated

BibTeX
@inproceedings{tortoreto2019affective,
  author = {Giuliano Tortoreto and Evgeny Stepanov and Alessandra Cervone and Mateusz Dubiel and Giuseppe Riccardi},
  title = {{Affective Behaviour Analysis of On-line User Interactions: Are On-line Support Groups More Therapeutic than Twitter?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure