Press Freedom Monitor: Detection of Reported Press and Media Freedom Violations in Twitter and News Articles

Tariq Yousef, Antje Schlaf, Janos Borst, Andreas Niekler, Gerhard HeyerEMNLP 2021

TL;DR

This paper introduces the "Press Freedom Monitor," an automated system designed to detect and flag reported press and media freedom violations in news articles and tweets. By leveraging deep learning models and a novel pipeline, it aims to assist organizations in their daily monitoring efforts and enable faster responses to threats against journalists.

What Problem Does This Paper Address?

Press freedom is under increasing threat globally, making it crucial to monitor and document violations. Existing manual monitoring processes are labor-intensive and can be slow to react. While some research has focused on detecting general online abuse or hate speech, there's a need for a specialized tool that can automatically identify and report specific incidents of press and media freedom violations, particularly in near real-time. This paper addresses the challenge of efficiently and accurately detecting these violations from vast amounts of online text data, such as news articles and tweets, to support organizations like the Media Freedom Rapid Response (MFRR).

Methods

The core of the Press Freedom Monitor is a deep learning-based classification pipeline. The process begins with data collection from Twitter and news APIs using a carefully curated set of keywords and hashtags related to press freedom violations and media actors. This is followed by duplicate detection using Locality Sensitive Hashing (LSH) to remove redundant content. Case mapping is then employed to group similar reports about the same incident, even if the wording differs, using semantic similarity based on text embeddings. A crucial component is the training phase, where the authors developed a custom dataset and evaluated various machine learning models. They found that deep learning models, specifically a Convolutional Neural Network (CNN) built on top of a distilled BERT model (DistilBERT), significantly outperformed traditional methods like Logistic Regression and Random Forests. The production pipeline integrates these components, including geocoding and country mapping to identify the location of incidents, and culminates in a user-friendly web interface for monitoring experts. Manual feedback from users is collected to further refine the model and prevent topic drift.

Datasets

The paper describes the creation of a custom training dataset for press and media freedom violations. This dataset was manually classified by experts from ECPMF, comprising 6,005 news articles and 8,192 tweets. The inter-annotator agreement was assessed and found to be substantial (Cohen's kappa of 0.62 for news and 0.63 for tweets). The training dataset consisted of 13,809 texts, with 90% used for training and 10% for testing. A separate set of 1,007 feedback items was used for validation. The authors note the absence of publicly available training datasets for this specific use case.

Key Insights and Ideas

  • Deep Learning Outperforms Traditional ML: The paper clearly demonstrates that deep learning models, particularly CNNs on transformer architectures, are significantly more effective than traditional machine learning approaches for this specific task. This highlights the power of contextual understanding in text classification.
  • The Importance of Custom Datasets: The lack of existing labeled data for press freedom violations necessitated the creation of a custom dataset. This underscores the challenge and importance of domain-specific data collection and annotation for specialized NLP tasks.
  • Near Real-Time Monitoring is Achievable: The system's architecture, with continuous data collection and processing, enables near real-time detection of violations, which is critical for rapid response.
  • Case Mapping for Context: Grouping similar reports about the same incident is a valuable feature, providing monitoring experts with a more comprehensive view and aiding verification.
  • Usability is Key for Adoption: The focus on a user-friendly interface and incorporating expert feedback is crucial for the practical adoption and effectiveness of such a tool in a professional setting.

Key Results

The CNN model achieved the highest average F1-score and recall during training and validation. Specifically, when evaluated on a confidence threshold of 0.8, the trained model achieved a recall of 0.87 and a precision of 0.96. Lowering the threshold to 0.5 maintained excellent precision (0.94) while increasing recall to 0.91. The system successfully identified 1.38% of news articles and 7.9% of tweets as duplicates. A usability study with 7 participants showed an overall satisfaction rate of 79%, with user interface quality scoring highest at 84%. The Hungarian language extension, despite using machine translation, proved valuable by detecting previously unknown incidents.

Limitations and Future Work

The authors acknowledge that the current system primarily focuses on English language content, with Hungarian being an initial extension. Future work includes:

  • Extending Language Support: Further expanding the system to monitor more languages.
  • Improving Manual Feedback Integration: Enhancing the use of manual feedback for retraining models and preventing topic drift.
  • Advanced Clustering for Case Mapping: Investigating clustering techniques to improve the grouping of similar incidents, especially in cases with high textual similarity.
  • Country Mapping for Distinction: Utilizing country mapping to better distinguish between incidents, potentially improving the cost-benefit ratio of analysis.

Summary updated

BibTeX
@inproceedings{yousef2021press,
  author = {Tariq Yousef and Antje Schlaf and Janos Borst and Andreas Niekler and Gerhard Heyer},
  title = {{Press Freedom Monitor: Detection of Reported Press and Media Freedom Violations in Twitter and News Articles}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
}
Figure