Aggression Detection in Social Media using Deep Neural Networks

Sreekanth Madisetty, Maunendra Sankar DesarkarCOLING 2018

TL;DR

This paper tackles the challenge of automatically detecting aggression in social media text, distinguishing between overtly aggressive, covertly aggressive, and non-aggressive content. They propose an ensemble of deep learning models (CNN, LSTM, Bi-LSTM) that achieves competitive results, particularly outperforming a random baseline.

What Problem Does This Paper Address?

The proliferation of user-generated content on social media platforms has unfortunately led to an increase in aggressive and harmful communication. Detecting this aggression is crucial for moderating online spaces, but it's a complex task. Aggression can manifest in various ways: directly and explicitly (overtly aggressive) or subtly and indirectly (covertly aggressive). Distinguishing between these forms, and from non-aggressive content, is particularly challenging, especially for covert aggression where the intent is hidden. Existing methods often struggle with this nuance, highlighting the need for more sophisticated detection techniques.

Methods

The authors propose an ensemble learning approach that combines three deep learning architectures:

  • Convolutional Neural Networks (CNNs): Inspired by Kim (2014), their CNN model uses five layers (input, convolution, pooling, hidden dense, and output) to extract features from text.
  • Long Short-Term Memory networks (LSTMs): LSTMs are employed to capture sequential dependencies in text, allowing them to remember long-term context, which is crucial for understanding nuanced language. The paper details the standard LSTM cell operations, including forget, input, and output gates.
  • Bi-directional Long Short-Term Memory networks (Bi-LSTMs): This architecture enhances LSTMs by processing the input sequence in both forward and backward directions, providing a richer contextual understanding.

These three models are then combined using a majority voting ensemble. For each input post, the final prediction is determined by the class predicted by the majority of the individual models. All deep learning models utilize GloVe word embeddings with a dimension size of 200.

Datasets

The study uses a dataset of Facebook comments. For training and validation, they utilize a dataset with 12,000 and 3,001 instances, respectively. For testing, they employ two distinct datasets:

  • Facebook posts: 916 instances (in-domain testing).
  • Social media posts: 1,257 instances (cross-domain testing). The data collection methods are described in Kumar et al. (2018b).

Key Insights and Ideas

  • Ensemble Power: The ensemble approach, combining CNN, LSTM, and Bi-LSTM, generally outperforms individual models, demonstrating the benefit of diverse feature extraction and prediction strategies.
  • Cross-Domain Robustness: The proposed method shows promising performance even when tested on social media posts from a different domain than the training data (Facebook), suggesting a degree of generalizability.
  • The Challenge of Covert Aggression: The results consistently show that covertly aggressive (CAG) content is the most difficult to classify accurately. This aligns with the inherent nature of covert aggression, where the intent is hidden and requires deeper contextual understanding.
  • Non-Aggressive is Easiest: The "Non-aggressive" (NAG) class consistently achieves the highest precision, recall, and F1-scores across both datasets. This is expected, as non-aggressive language typically lacks the subtle or overt cues that characterize aggressive content.

Key Results

  • Facebook Posts (In-domain): The proposed ensemble method achieved a weighted F1-score of 0.6037.
  • Social Media Posts (Cross-domain): The proposed ensemble method achieved a weighted F1-score of 0.5080.
  • Outperforming Baseline: In both in-domain and cross-domain evaluations, the proposed method significantly outperformed a random baseline, which achieved F1-scores of 0.3535 and 0.3477, respectively.
  • Confusion: The confusion matrices reveal that the most significant misclassifications occur between CAG and NAG, and between OAG and CAG.

Limitations and Future Work

The authors acknowledge that differentiating between covertly aggressive and non-aggressive posts remains a significant challenge. The performance on the Covertly Aggressive (CAG) class is lower compared to other classes, indicating room for improvement.

For future work, they suggest:

  • Hybrid Feature Engineering: Incorporating both feature-based methods (e.g., linguistic features) and deep learning methods within the ensemble.
  • Improving CAG Detection: Further research is needed to specifically address the challenges of accurately identifying covert aggression.

Summary updated

BibTeX
@inproceedings{madisetty2018aggression,
  author = {Sreekanth Madisetty and Maunendra Sankar Desarkar},
  title = {{Aggression Detection in Social Media using Deep Neural Networks}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
}
Figure