Weaving a Semantic Web of Credibility Reviews for Explainable Misinformation Detection (Extended Abstract)

Ronald Denaux, Martino Mensio, José Manuél Gómez-Pérez, Harith AlaniIJCAI 2021 · DOI

TL;DR

This paper introduces "Linked Credibility Reviews" (LCR), a novel framework that combines semantic web technologies with deep learning to create explainable misinformation detection systems. By building a network of distributed "bots" that collaboratively assess content credibility, the system aims to provide transparent and trustworthy insights into the veracity of online information.

What Problem Does This Paper Address?

The proliferation of misinformation online, amplified by social media and information bubbles, poses a significant societal challenge. While automated fact-checking systems exist, they often lack transparency, require custom integration for different tasks, and struggle to generalize to new data types. The authors argue that a decentralized, collaborative approach is needed, leveraging both symbolic knowledge (semantic web) and deep learning to build systems that not only detect misinformation but also explain why a piece of content is deemed credible or not, making them useful for both the general public and professional fact-checkers.

Methods

The core of the proposed solution is the "Linked Credibility Reviews" (LCR) model. This conceptual and data model defines a "Credibility Review" (CR) as a tuple containing the data item being reviewed, a credibility rating, a confidence score, and mandatory provenance information. This provenance details the signals and strategies used to arrive at the rating, enabling traceability and explainability.

The system employs a network of specialized "bots" that perform different review strategies:

  • LookupBots: Fetch existing "ground credibility signals" (e.g., ClaimReview data, external services like OpenSources).
  • LinkBots: Link content to similar, pre-crawled items with existing reviews, considering semantic similarity and stance.
  • DecomposeBots: Break down an item into smaller parts, request reviews for those parts, and then aggregate them.

These bots collaborate to build a "chained credibility review," where the output of one bot can serve as input for another. The system extends schema.org to represent these reviews and their provenance, making them interoperable with existing web data.

Datasets

The evaluation was conducted on three datasets:

  • CLEF'18 CheckThat! Factuality Task: An English dataset for predicting claim veracity (true, half-true, false).
  • Politifact fragment of FakeNewsNet: A dataset containing articles labeled as fake or real.
  • coInform250: A dataset of 250 annotated tweets with associated ClaimReviews.

Key Insights and Ideas

  • Credibility over Accuracy: The paper emphasizes focusing on "credibility" (an estimation of factuality based on available signals) rather than just raw accuracy, acknowledging the subjective nature of truth assessment.
  • Decentralization and Collaboration: The framework promotes a decentralized approach where multiple bots with different strategies can collaborate, mimicking a distributed fact-checking network.
  • Explainability through Provenance: The mandatory inclusion of provenance information is crucial for building trust and allowing users to understand how a credibility assessment was made.
  • Extending Semantic Web Standards: Leveraging and extending schema.org provides a standardized way to represent credibility reviews, fostering interoperability.

Key Results

The proposed system, "acred," achieved state-of-the-art results on the CLEF'18 CheckThat! Factuality Task, with a further improvement using a modified version, "acred+." On the FakeNewsNet dataset, "acred+" also achieved state-of-the-art results, while "acred" showed competitive performance against strong baselines. For the coinform250 dataset, "acred+" achieved a significantly better accuracy than random prediction. The system demonstrated its ability to produce human-understandable explanations.

Limitations and Future Work

The authors acknowledge that errors in low-level modules (like stance detection) can impact the overall assessment, and they are exploring automated ways to correct such errors. A caveat mentioned is that many datasets are derived from fact-checking articles, which might introduce noise. Future work includes researching how the generated explanations and evidence graphs can be used to gather user feedback for pinpointing error sources.

Summary updated

BibTeX
@inproceedings{denaux2021weaving,
  author = {Ronald Denaux and Martino Mensio and José Manuél Gómez-Pérez and Harith Alani},
  title = {{Weaving a Semantic Web of Credibility Reviews for Explainable Misinformation Detection (Extended Abstract)}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/646},
}
Figure