Forecasting the presence and intensity of hostility on Instagram using linguistic and social features

Ping Liu, Joshua Guberman, Libby Hemphill, Aron Culotta

Abstract: Online antisocial behavior, such as cyberbullying, harassment, and trolling, is a widespread problem that threatens free discussion and has negative physical and mental health consequences for victims and communities. While prior work has proposed automated methods to identify hostile comments in online discussions, these methods work retrospectively on comments that have already been posted, making it difficult to intervene before an interaction escalates. In this paper we instead consider the problem of forecasting future hostilities in online discussions, which we decompose into two tasks: (1) given an initial sequence of non-hostile comments in a discussion, predict whether some future comment will contain hostility; and (2) given the first hostile comment in a discussion, predict whether this will lead to an escalation of hostility in subsequent comments. Thus, we aim to forecast both the presence and intensity of hostile comments based on linguistic and social features from earlier comments. To evaluate our approach, we introduce a corpus of over 30K annotated Instagram comments from over 1,100 posts. Our approach is able to predict the appearance of a hostile comment on an Instagram post ten or more hours in the future with an AUC of .82 (task 1), and can furthermore distinguish between high and low levels of future hostility with an AUC of .91 (task 2).

1. Introduction

Online safety research has traditionally focused on detection—the retrospective identification of abuse after it has occurred. However, by the time a system flags a toxic comment, the harm to the victim has already been done, and the conversation may have already spiraled into a conflict that is difficult to de-escalate. This paper, presented at ICWSM 2018, proposes a shift toward forecasting: predicting the arrival and intensity of hostile content hours before it appears by analyzing early linguistic signals and social context.

2. Problem Formulation

The researchers decompose the forecasting problem into two distinct binary classification tasks:

  1. Hostility Presence: Given a sequence of initial non-hostile comments, will at least one hostile comment appear in the future?
  2. Hostility Intensity: Given the arrival of the first hostile comment, will the thread escalate to a high level of hostility (defined as $N$ or more subsequent hostile comments)?

3. Dataset and Annotation

The study introduces a new, manually annotated corpus specifically designed for temporal analysis. The dataset consists of 30,987 comments from 1,134 Instagram posts, sampled from public accounts likely belonging to U.S. teenagers. Unlike standard toxicity datasets, this corpus preserves the full temporal sequence of comments.

Distribution of comments, users, and temporal features across the Instagram dataset.

As shown in the figure above, the dataset exhibits a long-tail distribution across several metrics. While the median number of comments per post is 14, some posts attract hundreds. Crucially, the median time until the first hostile comment is just 1 day, though the mean is 13 days, indicating that while some conflicts ignite quickly, others brew over longer periods.

4. Methodology

The authors utilized Logistic Regression with L2 regularization and engineered four primary feature sets:

  • Linguistic Features: Unigrams and FastText-style n-gram Word2vec embeddings to capture the nuances of social media slang and typos.
  • Trend Features: A novel set of features derived from a separate hostility detector. By calculating the "slope" and "variance" of hostility probabilities over the initial comment sequence, the model identifies if a conversation is "heating up."
  • Social Context: Features including "Prev-post" (the historical hostility levels of the photo's author) and "Prev-com" (the history of the current commenters).
  • User Activity: Metrics such as the ratio of unique users to total comments and the frequency of @mentions.

5. Key Results and Insights

The models demonstrated high predictive power across both tasks:

  • Task 1 (Presence): The model achieved an AUC of 0.82 when forecasting hostility 10 or more hours into the future. Performance was remarkably stable; the AUC only dropped from 0.84 (1-hour lead time) to 0.82 (10-hour lead time), suggesting that the trajectory of a thread is often established very early.
  • Task 2 (Intensity): The model excelled at predicting escalation, reaching an AUC of 0.91 when distinguishing between isolated insults and major harassment campaigns.

The "Magnet" Effect

A significant finding was the importance of the post author's history ("Prev-post"). If an author's previous content frequently attracted hostility, their new posts were much more likely to become targets again, regardless of the initial tone of the new thread. Conversely, the history of current commenters ("Prev-com") was less useful, as the first hostile actor is often a stranger who has not yet participated in the discussion.

6. Conclusion

This work demonstrates that hostility on social media is not entirely random but follows predictable linguistic and social patterns. While "non-sequitur hostility"—sudden, unrelated insults—remains a challenge, the ability to forecast conflict spirals with high accuracy opens new doors for proactive moderation and victim protection strategies.

Summary updated

Abstract

Online antisocial behavior, such as cyberbullying, harassment, and trolling, is a widespread problem that threatens free discussion and has negative physical and mental health consequences for victims and communities. While prior work has proposed automated methods to identify hostile comments in online discussions, these methods work retrospectively on comments that have already been posted, making it difficult to intervene before an interaction escalates. In this paper we instead consider the problem of forecasting future hostilities in online discussions, which we decompose into two tasks: (1) given an initial sequence of non-hostile comments in a discussion, predict whether some future comment will contain hostility; and (2) given the first hostile comment in a discussion, predict whether this will lead to an escalation of hostility in subsequent comments. Thus, we aim to forecast both the presence and intensity of hostile comments based on linguistic and social features from earlier comments. To evaluate our approach, we introduce a corpus of over 30K annotated Instagram comments from over 1,100 posts. Our approach is able to predict the appearance of a hostile comment on an Instagram post ten or more hours in the future with an AUC of .82 (task 1), and can furthermore distinguish between high and low levels of future hostility with an AUC of .91 (task 2).
BibTeX
@inproceedings{liu2018forecasting,
  author = {Ping Liu and Joshua Guberman and Libby Hemphill and Aron Culotta},
  title = {{Forecasting the presence and intensity of hostility on Instagram using linguistic and social features}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/icwsm.v12i1.15022},
}

Loading related content...

Figure