Investigating radicalisation indicators in online extremist communities

Christine De Kock, Eduard HovyNAACL 2024

TL;DR

This paper introduces a novel framework for identifying early indicators of radicalization in online extremist communities by analyzing hostility, longevity, and social connectivity. Their NLP-based models can predict users at risk with promising accuracy, offering practical insights for intervention strategies.

What Problem Does This Paper Address?

Online extremism is a growing concern, and while there's a wealth of publicly available content, effectively detecting and combating extremist behavior remains a challenge. Existing Natural Language Processing (NLP) research has largely focused on specific communities or surface-level features, with a dearth of work on the more general process of radicalization. A key difficulty is the ill-defined nature of "radicalization," often treated as a binary state rather than a gradual process. This paper aims to address this by identifying and quantifying observable behaviors that indicate radicalization at an individual level within extremist online communities.

Methods

The authors propose a holistic approach by considering three key indicators of radicalization:

  • Hostility: Measured using a lexicon of in-group language specific to extremist communities, reflecting adoption of radical ideas and support for violent acts.
  • Longevity: Quantified by the total number of posts a user makes on the platform, indicating sustained engagement with the doctrine.
  • Social Connectivity (Centrality): Assessed using betweenness centrality on an interaction graph, measuring a user's importance within the community network.

They developed and evaluated several NLP models, including:

  • Feature-based models: Utilizing engineered features like hostility terms and post counts.
  • Text-based models: Employing transformer embeddings (all-mpnet-base-v2) to capture nuanced linguistic features.
  • Mixed-input models: Combining both engineered features and text embeddings.
  • Multitask models: Training a single model to predict all three indicators simultaneously, leveraging parameter sharing.

They also explored survival regression for the longevity prediction task, which is more suitable for time-to-event data.

Datasets

The primary dataset used is from Ribeiro et al. (2021), comprising 7.4 million posts from 139,090 users across 8 extremist anti-women forums between 2005 and 2019. For evaluating generalizability, they also used the ExtremeBB dataset from Vu et al. (2021), which includes posts from the white supremacy platform Stormfront.

Key Insights and Ideas

  • Complementary Indicators: Hostility, longevity, and centrality provide distinct yet interacting signals of radicalization. While correlated, each offers a unique perspective, with hostility being the most distinct.
  • Early Churn is Common: Many users quickly disengage from extremist forums, suggesting that initial exposure might not resonate with their beliefs.
  • Varied Paths to Hostility: Some users exhibit hostility from the outset, while others develop it over time, highlighting the spectrum of radicalization.
  • Early Signals Matter: Specific early behaviors, like the number of distinct days a user engages, are predictive of eventual radicalization indicators, even if not correlated with each other.
  • Multitask Learning is Beneficial: Jointly predicting the three indicators improves performance, suggesting they are mutually informative.

Key Results

The study demonstrates that NLP models can effectively predict radicalization indicators.

  • Prediction Accuracy: The top-performing multitask model achieved a concordance index (CI) of 0.667 for centrality, 0.698 for hostility, and 0.681 for longevity at N=10 posts. This represents a statistically significant improvement over baseline models.
  • Early Prediction: The models show promising predictive power even with a small number of initial posts (as few as 5-6 posts), offering a good tradeoff between early detection and accuracy.
  • Model Performance: Text-based models (embeddings) outperformed feature-based approaches for hostility and longevity, while mixed-input and multitask models generally yielded the best results.
  • Cross-Community Transfer: While performance drops when models trained on one community are applied to another, auxiliary tasks like forum prediction can mitigate this.

Limitations and Future Work

The authors acknowledge several limitations:

  • Lexicon Dependency: The hostility indicator relies on lexicons, which can be imperfect, dynamic, and community-specific.
  • Centrality Limitations: The centrality metric may not fully capture social connectedness in cases of community splintering.
  • Churn Interpretation: The assumption that early churn is solely due to disengagement might overlook users seeking more extreme groups.
  • Community Specificity: Further work is needed to understand characteristics that differentiate extremist communities.

Future work could involve incorporating more diverse data sources, exploring additional behavioral and psychological parameters, and developing more robust methods for capturing dynamic language evolution within communities.

Summary updated

BibTeX
@inproceedings{kock2024investigating,
  author = {Christine De Kock and Eduard Hovy},
  title = {{Investigating radicalisation indicators in online extremist communities}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure