Mapping (Dis-)Information Flow about the MH17 Plane Crash

Mareike Hartmann, Yevgeniy Golovchenko, Isabelle AugensteinarXiv 2019 · arXiv

Introduction

The 2014 shoot-down of Malaysian Airlines flight MH17 over Ukraine initiated a complex information war. For political scientists, understanding the spread of disinformation requires labeling thousands of social media posts as either pro-Russian (attributing blame to Ukraine or NATO) or pro-Ukrainian (attributing blame to Russia or separatists). Manual annotation is prohibitively slow, and traditional "distant supervision" methods—such as labeling based on hashtags—often prove too noisy for rigorous analysis. This study investigates the efficacy of modern Natural Language Processing (NLP) techniques, specifically Convolutional Neural Networks (CNNs), in automatically categorizing these competing narratives.

Methodology

Researchers compared three distinct classification approaches to categorize tweets into Pro-Russian, Pro-Ukrainian, or Neutral classes:

  1. Hashtag-Based Baseline (HS_PMI): A simple model using Pointwise Mutual Information to associate specific hashtags with classes.
  2. Logistic Regression (LOGREG): A standard baseline utilizing pre-trained fastText word embeddings.
  3. Convolutional Neural Network (CNN): A 1D-CNN architecture using fastText embeddings.

The study also explored cross-lingual data augmentation, attempting to transfer knowledge from Russian-language "troll" accounts to English datasets using aligned word embeddings. However, this strategy did not yield significant improvements, likely because the troll data lacked the specific technical nuances of the MH17 investigation.

The "Neutral" Gravity Well and Classification Challenges

A significant challenge in this domain is the extreme class imbalance. The vast majority of tweets are "Neutral," consisting of standard news reports or expressions of grief. This creates a "gravity well" where models can achieve high overall accuracy simply by over-predicting the neutral class, while failing to identify the rare but critical polarized content.

CNN Confusion Matrix

As shown in the confusion matrix, while the CNN model identifies neutral tweets with 90% accuracy, its performance on polarized frames is lower (57% for Pro-Russian and 71% for Pro-Ukrainian). Errors typically fall into three categories: simple text misses, requirements for event-specific technical knowledge (e.g., weapon systems like the "SU25"), and the inability to parse irony or sarcasm.

Performance and the Precision-Recall Trade-off

The CNN was the most effective model, achieving a Macro-F1 of 0.69, significantly outperforming Logistic Regression (0.59) and the hashtag baseline (0.25). However, for scientific analysis requiring high reliability (e.g., 80% precision), the model's recall drops to approximately 23%.

CNN Precision-Recall Curve

This trade-off indicates that while the model is excellent at finding "sure bets," it misses a large portion of polarized content. Despite these limitations, the classifier serves as a powerful "force multiplier." By using the model to filter tweets for human review, researchers increased the identification of pro-Russian edges in retweet networks by 29% compared to manual efforts alone.

Conclusion

While fully automated high-precision labeling remains elusive due to the need for technical context and irony detection, NLP models like CNNs provide a substantial advantage over keyword-based methods. Future work should focus on incorporating external knowledge bases and network-aware features to bridge the current knowledge gap in disinformation tracking.

Summary updated

Abstract

Digital media enables not only fast sharing of information, but also disinformation. One prominent case of an event leading to circulation of disinformation on social media is the MH17 plane crash. Studies analysing the spread of information about this event on Twitter have focused on small, manually annotated datasets, or used proxys for data annotation. In this work, we examine to what extent text classifiers can be used to label data for subsequent content analysis, in particular we focus on predicting pro-Russian and pro-Ukrainian Twitter content related to the MH17 plane crash. Even though we find that a neural classifier improves over a hashtag based baseline, labeling pro-Russian and pro-Ukrainian content with high precision remains a challenging problem. We provide an error analysis underlining the difficulty of the task and identify factors that might help improve classification in future work. Finally, we show how the classifier can facilitate the annotation task for human annotators.
BibTeX
@misc{hartmann2019mapping,
  author = {Mareike Hartmann and Yevgeniy Golovchenko and Isabelle Augenstein},
  title = {{Mapping (Dis-)Information Flow about the MH17 Plane Crash}},
  year = {2019},
  eprint = {1910.01363},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.01363},
}
Figure