Team QCRI-MIT at SemEval-2019 Task 4: Propaganda Analysis Meets Hyperpartisan News Detection

Abdelrhman Saleh, Ramy Baly, Alberto Barrón-Cedeño, Giovanni Da San Martino, Mitra Mohtarami, Preslav Nakov, James R. Glass

Abstract

Hyperpartisan news detection is a critical challenge in combating disinformation. This study investigates the hypothesis that hyperpartisanship shares stylistic similarities with propaganda. By employing a logistic regression model integrated with diverse features—including n-grams, bias lexicons, vocabulary richness, and readability scores—the researchers achieved 72.9% accuracy on manually labeled articles. The results highlight a significant performance gap between models trained on clean, article-level data versus noisy, publisher-level labels, suggesting that stylistic features are highly sensitive to label quality.

Introduction

The rise of social media has facilitated the rapid spread of "fake news" and hyperpartisan content—news characterized by a one-sided, often blind allegiance to a political cause. Detecting such content is difficult because hyperpartisan articles often cover the same topics as mainstream news, rendering simple keyword-based filters ineffective. This research posits that hyperpartisanship is defined not just by what is said, but by how it is said. By treating hyperpartisanship as a sibling to propaganda, the authors utilize stylometric features to distinguish biased reporting from objective journalism.

Methodology

The authors developed a binary classification system using Logistic Regression based on several feature families:

  • N-gram Analysis: The model incorporates character 3-grams to capture stylistic markers and word 1,2-grams. These are weighted using NB-TF.IDF, which scales frequencies based on their importance for classification using Naive Bayes ratios.
  • Bias Lexicons: Automatically generated lexicons based on Semantic Orientation (SO) were used to calculate bias intensity scores for "left" or "right" leanings.
  • Vocabulary Richness: Stylometric fingerprints were captured through metrics such as the Type-Token Ratio, Hapax Legomena, Honore’s R, and Yule’s K.
  • Readability and Complexity: Structural complexity was measured using standard tests like the Flesch-Kincaid grade level and the Gunning fog index.
  • Linguistic Lexicons: Frequency counts from 18 established lexicons (e.g., swear words, hedges, negations) provided additional stylistic context.

Datasets

The system was evaluated using the SemEval-2019 Task 4 Dataset, which comprises two distinct subsets:

  1. Labeled by-Publisher: A large-scale "distant supervision" dataset of 750,000 articles where labels are assigned based on the publisher's general reputation.
  2. Labeled by-Article: A "gold standard" set of 645 articles manually labeled by humans, providing a high-quality benchmark for evaluation.

Results and Analysis

The experimental results demonstrate a clear link between propaganda detection and hyperpartisan identification. Character-level n-grams proved particularly effective as they are topic-agnostic, focusing on writing style rather than subject matter.

A critical finding was the impact of feature engineering on different labeling schemes. As shown in Table 1, adding sophisticated features like vocabulary richness and readability consistently improved accuracy on the manually labeled "by-article" data. However, these same features often led to a performance decline on the "by-publisher" data.

Table 1: Performance metrics across different feature combinations for article-level and publisher-level labels.

This discrepancy suggests that distant supervision introduces significant noise; an article from a generally biased publisher may still be written in a neutral style, confusing models that rely on fine-grained stylometric markers. In the official competition, the team's submission (using word and character n-grams) reached 72.9% accuracy on the manual test set, ranking 20th out of 42 teams. Post-competition refinements in feature scaling further increased this accuracy to 76.0%.

Conclusion

The study confirms that hyperpartisanship and propaganda are stylistically related. While complex linguistic features significantly enhance detection on high-quality, manually labeled data, they are susceptible to the noise found in large-scale, publisher-level datasets. Future work should focus on improving feature pre-processing and exploring deep learning architectures that can maintain interpretability while capturing the nuances of linguistic style.

Summary updated

Abstract

In this paper, we describe our submission to SemEval-2019 Task 4 on Hyperpartisan News Detection. Our system relies on a variety of engineered features originally used to detect propaganda. This is based on the assumption that biased messages are propagandistic in the sense that they promote a particular political cause or viewpoint. We trained a logistic regression model with features ranging from simple bag-of-words to vocabulary richness and text readability features. Our system achieved 72.9% accuracy on the test data that is annotated manually and 60.8% on the test data that is annotated with distant supervision. Additional experiments showed that significant performance improvements can be achieved with better feature pre-processing.
BibTeX
@inproceedings{saleh2019team,
  author = {Abdelrhman Saleh and Ramy Baly and Alberto Barrón-Cedeño and Giovanni Da San Martino and Mitra Mohtarami and Preslav Nakov and James R. Glass},
  title = {{Team QCRI-MIT at SemEval-2019 Task 4: Propaganda Analysis Meets Hyperpartisan News Detection}},
  booktitle = {SemEval @ NAACL},
  year = {2019},
  doi = {10.18653/v1/s19-2182},
}
Figure