Journalist-in-the-Loop: Continuous Learning as a Service for Rumour Analysis

Twin Karmakharm, Nikolaos Aletras, Kalina BontchevaEMNLP 2019 · OpenReview

Abstract

Twin Karmakharm, Nikolaos Aletras, Kalina Bontcheva. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations.

2019.

Introduction

In the high-stakes world of journalism, identifying and debunking rumours on social media is a race against time. While machine learning (ML) models can help scale this process, they suffer from a "static model" problem: a model trained on historical data often fails when faced with the unique linguistic patterns of a brand-new event. Currently, journalists use a fragmented toolkit—Google reverse image search, Tweetdeck, and various experimental algorithms—but these tools are usually "black boxes" that do not learn from the journalist's expertise. This research introduces a "Continuous Learning as a Service" platform that bridges this gap by putting the journalist directly in the loop.

System Architecture

The platform is built on three core pillars designed to facilitate a continuous feedback loop between human experts and machine learning models.

The system architecture showing the continuous learning loop between the classification system and the human annotation service.
  1. Rumour Classification System: A PyTorch-based neural network that categorizes tweets as true, false, or unverified. The architecture uses a Long Short-Term Memory (LSTM) network combined with an Inner-Attention Mechanism. It processes three inputs: the source tweet text, "recurring terms" (words historically associated with specific veracity categories), and "stance" (the proportion of replies that support, deny, or question the original post).
  2. Rumour Annotation Service: A user-friendly GUI and REST API where journalists can quickly label the veracity of a rumour or the stance of individual replies.
  3. Continuous Learning Loop: Unlike traditional models that are trained once and deployed, this system is designed to be retrained regularly using the new, journalist-provided labels stored in its database.

The Human-in-the-Loop Interface

For continuous learning to be effective in a newsroom, the "manual overhead" for journalists must be minimal. The UI is designed to allow annotation as a natural side effect of the journalist's existing verification workflow.

The web-based GUI allows journalists to view tweet metadata, assess veracity on a scale, and vote on the stance of replies.

Journalists can provide feedback on the veracity of a tweet using a sliding scale and categorize the stance of replies (Support, Comment, Question, Deny). To handle potential human error or bias, the system uses a majority vote (requiring at least 50% agreement) when multiple journalists annotate the same piece of content before using it for retraining.

Data and Performance

The model was initially "seeded" using the RumourEval2017 dataset, derived from the PHEME dataset. This contains 325 Twitter conversation threads covering eight major events, such as the Germanwings air crash and the Charlie Hebdo shooting.

In system evaluations, the underlying "Inner-Attention" algorithm achieved an F1 score of 0.616 and an accuracy of 0.607 on the RumourEval2017 test set. This significantly outperformed several state-of-the-art approaches, including vanilla LSTMs and multi-task learning models, which hovered between 0.49 and 0.57 in F1 scores. The majority class baseline (simply guessing the most common category) was only 0.429, demonstrating the model's substantial value.

Key Insights and Future Directions

  • The Power of Small Data: Machine learning models do not always need massive datasets to improve. Previous research integrated into this project suggests that just ten human-labelled examples from a newly emerging rumour can improve algorithmic accuracy by at least 10%.
  • Stance as a Proxy for Truth: The system doesn't just look at what a tweet says; it looks at how the "crowd" reacts. By calculating the proportions of supporting vs. denying comments, the model gains a social context that raw text analysis often misses.
  • Future Work: While the current prototype is focused on Twitter, the authors acknowledge the need to expand to platforms like Reddit and 4chan. Future iterations also aim to integrate "user credibility" scores to weight annotations from trusted journalists more heavily and provide deeper propagation visualizations.

Summary updated

BibTeX
@inproceedings{karmakharm2019journalist,
  author = {Twin Karmakharm and Nikolaos Aletras and Kalina Bontcheva},
  title = {{Journalist-in-the-Loop: Continuous Learning as a Service for Rumour Analysis}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=kJmEJCd5jvt},
}
Figure