Towards a Multi-Entity Aspect-Based Sentiment Analysis for Characterizing Directed Social Regard in Online Messaging

Joan Zheng, Scott Friedman, Sonja Schmer-galunder, Ian Magnusson, Ruta Wheelock, Jeremy Gottlieb, Diana Gomez, Christopher MillerNAACL 2022

TL;DR

This paper introduces Directed Social Regard (DSR), a novel framework for analyzing multi-entity, multi-sentiment online messages. By modeling sentiment intensity across nine psychosocial dimensions, they offer a more nuanced understanding of online discourse, particularly in contexts of abuse and hate speech, moving beyond simple positive/negative classifications.

What Problem Does This Paper Address?

Online communication is complex: a single message can express conflicting sentiments towards multiple individuals or groups. Traditional sentiment analysis often simplifies this by assigning a single polarity to an entire message or a single aspect. This fails to capture the intricate dynamics of online interactions, such as simultaneously condemning one party while empathizing with another, or the subtle ways people express advocacy, opposition, or dehumanization. The authors aim to address this gap by developing a system that can quantify the directed social regard—the specific attitudes and their intensities—towards multiple entities within a single message, drawing inspiration from theories of moral disengagement.

Methods

The core contribution is a Multi-Entity Aspect-Based Sentiment Analysis (ME-ABSA) approach, specifically tailored for Directed Social Regard (DSR). Instead of just classifying sentiment as positive/negative/neutral, their model predicts the intensity of sentiment along nine continuous dimensions: advocate, oppose, dehumanization, empathy, violent, condemn, justified, responsible (for harm), and harmed.

Key methodological aspects include:

  • Knowledge Graph Representation: Entities (characters, events) are identified and structured, forming a knowledge graph.
  • Transformer-based Regression: They leverage a transformer architecture (specifically, a fine-tuned BERT model) not for classification, but for regression. This allows for fine-grained, continuous intensity scores for each sentiment dimension.
  • Masked Language Modeling: Used to support the aspect-specific scoring of entities.
  • Entity Masking: A mechanism to ensure that only relevant entities are scored for specific aspects (e.g., an "event" entity wouldn't be scored for "dehumanization").

Datasets

The authors created a new dataset by annotating posts from various sources known for online abuse and hate speech, including:

  • Moral Foundations Twitter Corpus
  • Gab Hate Corpus
  • How ISIS Uses Twitter dataset
  • Manosphere community text posts

This dataset comprises over 2,900 posts and sentences, with more than 24,000 entities annotated for DSR across the nine dimensions by three human annotators. Annotations were on a 0-5 intensity scale.

Key Insights and Ideas

  • Beyond Polarity: The most significant insight is the necessity of moving beyond simple sentiment polarity to a multi-dimensional, intensity-based approach to truly understand online social dynamics, especially in sensitive contexts.
  • Moral Disengagement as a Framework: Leveraging Bandura's theory of moral disengagement provides a strong theoretical grounding for the chosen sentiment dimensions, offering a psychological lens on why people express certain attitudes.
  • Nuance in Online Discourse: The ability to capture conflicting sentiments within a single message (e.g., "I condemn X but empathize with Y") is crucial for accurate analysis of online discourse.
  • The Power of Regression: Using regression instead of classification allows for a more granular understanding of sentiment intensity, which is vital for measuring the degree of advocacy, opposition, or dehumanization.

Key Results

  • The model achieved promising preliminary results, with an average RMSE of 1.00 on a 5-point intensity scale across all aspects.
  • The lowest error (RMSE) was observed for the "violent," "dehumanized," and "empathy" dimensions, which the authors attribute to these aspects being more intensely rated and having higher inter-annotator agreement (Krippendorff's α).
  • The model's R² values indicate it explains between 11-29% of the variance in human annotators' intensity scores.
  • The entity extractor component achieved high F1 scores (0.95 for characters, 0.73 for events) on a 90/10 train/test split.

Limitations and Future Work

  • Imbalanced Data: The dataset suffers from an imbalanced representation of sentiments, with a scarcity of examples for moderate to positive sentiments and low-to-moderate intensity levels. This also led to lower inter-annotator agreement for aspects like "empathy" and "justified."
  • Limited Agreement on Certain Aspects: Annotators struggled to reach reliably high agreement on sentiments like "empathy" and "justified," suggesting these are harder to define and consistently label.
  • Future Work:
    • Expand the dataset to include a more diverse range of sentiments (positive, neutral) and intensity levels to improve model performance.
  • Apply the model to broader domains beyond online abuse and hate speech.
  • Further refine the ME-ABSA architecture for even more accurate DSR prediction.

Summary updated

BibTeX
@inproceedings{zheng2022towards,
  author = {Joan Zheng and Scott Friedman and Sonja Schmer-galunder and Ian Magnusson and Ruta Wheelock and Jeremy Gottlieb and Diana Gomez and Christopher Miller},
  title = {{Towards a Multi-Entity Aspect-Based Sentiment Analysis for Characterizing Directed Social Regard in Online Messaging}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure