POLygraph: Polish Fake News Dataset

Daniel Dzienisiewicz, Filip Graliński, Piotr Jabłoński, Marek Kubis, Paweł Skórzewski, Piotr WierzchonACL 2024

TL;DR

This paper introduces POLygraph, a comprehensive dataset for detecting fake news in Polish. It addresses the growing challenge of misinformation by providing a rich resource that goes beyond simple binary classification, incorporating nuanced annotations and diverse data sources to foster more robust fake news detection models.

What Problem Does This Paper Address?

The proliferation of fake news poses a significant threat to public trust, opinion manipulation, and societal stability. While numerous efforts have been made to combat this issue, a critical gap exists in high-quality, nuanced datasets for languages other than English. This paper tackles the challenge of detecting and analyzing fake news specifically within the Polish digital landscape, a region underserved by existing resources. The authors highlight that fake news detection is complex, involving not just the content itself but also the sources, disseminators, and the social context surrounding its spread.

Methods

The POLygraph dataset was constructed using a hybrid approach combining API data access and web scraping. For Twitter (now X), the Twitter API was utilized to access archived data. For websites, a custom scraper was developed to extract relevant content. The data collection involved several stages:

  • Source Identification: A list of 1,300 starter websites was scraped to gather a broad range of content.
  • URL Extraction: A sophisticated XPath expression was used to extract links from documents, with specific exclusions for mailto and tel links.
  • Content Archiving: Scraped pages were archived as HTML files and compressed into ZIP archives, anonymized with a 128-bit hash of the URL.
  • Tweet Analysis: Tweets were queried using specific Polish and English phrases related to fake news and fact-checking to identify discussions about article truthfulness.
  • URL Filtering: A series of filters were applied to the extracted URLs to remove irrelevant or redundant entries, including those without external URLs, in other languages, with multiple links, or from fact-checking websites.
  • Screenshot Generation: A Python spider was developed to capture screenshots of both tweets and the articles they linked to, creating visual context for annotators.
  • Anonymization: A three-module tool was developed for anonymization, including a named entity recognizer, an alphanumeric expression classifier, and a privacy checker, to remove potentially private information while retaining public figures' names.

Datasets

The paper introduces the POLygraph dataset, which consists of two main parts:

  • "fake-or-not" dataset: Contains 11,360 pairs of news articles (identified by URLs) with labels indicating whether the news is fake or not. This dataset was built from a pilot annotation phase, extensive annotation tasks on the Doccano platform, and data from fact-checking websites.
  • "fake-they-say" dataset: Comprises 5,082 news articles (identified by URLs) and associated tweets commenting on them. Each tweet is labeled with the commentator's opinion on the article's truthfulness, using six distinct labels (hard-claim-fake, hard-claim-not-fake, no-claim, sarcasm, soft-claim-fake, soft-claim-not-fake). This dataset was also derived from pilot annotation and Doccano annotation tasks.

Key Insights and Ideas

  • Beyond Binary Classification: The POLygraph dataset moves beyond a simple true/false classification by incorporating multiple layers of annotation, including the type of text, subject matter, annotator's attitude and emotions, dominant content (facts vs. opinions), persuasiveness, purpose of the news, potential target, author's belief, source citation, narrative style, and the nature and impact of false information. This richer annotation scheme allows for a more nuanced understanding of fake news.
  • Multidisciplinary Approach: The project involved an interdisciplinary team of experts from various fields (mathematics, computer science, philology, media studies, law, philosophy, folklore, IT), highlighting the complex nature of fake news and the need for diverse perspectives in its detection.
  • Importance of Context: The dataset emphasizes the importance of context, not just in the article content but also in the surrounding tweets and the origin of the information, which are crucial for effective fake news detection.
  • Adaptability for Other Languages: The authors suggest that the underlying principles and annotation methodologies of POLygraph can be adapted for other languages, promoting universality in fake news detection research.

Key Results

The paper primarily focuses on the creation and composition of the dataset. Specific quantitative results are presented in terms of dataset sizes:

  • "fake-or-not" dataset: 11,360 article pairs.
  • "fake-they-say" dataset: 5,082 article-tweet pairs.

The paper also reports inter-annotator agreement scores:

  • Fake-or-not: Cohen's kappa of 0.421 for the pilot annotation and Fleiss' kappa varying by question for the main annotation.
  • Fake-they-say: Fleiss' kappa of 0.4343 for the annotation tasks.

These scores, while not perfect, are deemed acceptable and comparable to similar studies in the field, suggesting the dataset's usefulness.

Limitations and Future Work

  • Generalizability: The dataset, while extensive, might not capture every form of misinformation online, potentially limiting the generalizability of findings.
  • Subjectivity in Annotation: Human annotation inherently introduces subjectivity, and differing definitions of "fake news" among annotators can lead to disagreements, especially in borderline cases.
  • Focus on Polish Language: The study's primary focus on the Polish language limits its direct applicability to other linguistic contexts.
  • Dynamic Nature of Fake News: The ever-evolving tactics of fake news require continuous updates to the dataset and detection tools.
  • Future Work: The paper states that a detailed evaluation of the software tool for content authenticity analysis is planned for a later stage. Further exploration of the collected data by an interdisciplinary team is expected to foster fake news detection and provide institutions with a more comprehensive resource. The authors also envision adapting the dataset for other languages.

Summary updated

BibTeX
@inproceedings{dzienisiewicz2024polygraph,
  author = {Daniel Dzienisiewicz and Filip Graliński and Piotr Jabłoński and Marek Kubis and Paweł Skórzewski and Piotr Wierzchon},
  title = {{POLygraph: Polish Fake News Dataset}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure