A Language-Based Approach to Fake News Detection Through Interpretable Features and BRNN

Yu Qiao, Daniel Wiechmann, Elma KerzUvA-DARE 2020

TL;DR

This paper introduces a novel approach to fake news detection that leverages interpretable linguistic features and Bi-directional Recurrent Neural Networks (BRNNs). The authors demonstrate that their "white-box" models achieve performance comparable to state-of-the-art "black-box" models, offering a more transparent and explainable solution for identifying misinformation.

What Problem Does This Paper Address?

The proliferation of "fake news" poses a significant threat to informed public discourse and trust in established media. While current machine learning models can detect fake news with high accuracy, they often rely on "black-box" approaches, such as word embeddings, which lack transparency. This makes it difficult to understand why a piece of news is flagged as fake, hindering trust and the ability to improve these systems. There's a growing demand for "white-box" or explainable models, especially in critical domains like news, to provide insights and build confidence. This paper aims to bridge this gap by developing a fake news detection system that is both accurate and interpretable.

Methods

The core of the proposed method involves training BRNN classifiers on a set of human-interpretable linguistic features. These features are extracted using a computational tool called CoCoGen, which employs a sliding window technique to capture "local" distributions of feature scores within a text. CoCoGen analyzes text across six categories: syntactic complexity, lexical richness, register-based n-gram frequency, information theory, LIWC-style features, and word-prevalence measures. For the LIAR dataset, which includes metadata like speaker profiles and context, the authors extend their BRNN architecture to incorporate these additional features. They also explore an ordinal classification approach for the LIAR dataset, treating the truthfulness labels as ordered categories rather than distinct classes.

Datasets

The study utilizes two benchmark datasets for fake news detection:

  • ISOT Dataset: A large dataset of over 40,000 real and fake news articles, primarily focusing on political and world news. It provides article titles and text.
  • LIAR Dataset: A dataset of over 12,000 short statements (claims) collected from various online sources. This dataset is unique in that it provides a six-level ordinal classification of truthfulness and includes rich metadata about the speaker (party affiliation, job, home state).

The paper does not introduce new benchmark datasets but rather uses existing ones to evaluate their proposed approach.

Key Insights and Ideas

  • Interpretable Features Can Compete with Black Boxes: The research strongly suggests that carefully selected, human-interpretable linguistic features, when combined with powerful models like BRNNs, can achieve performance on par with state-of-the-art black-box models that rely on opaque word embeddings. This is a significant finding for the field, as it opens the door to more transparent and trustworthy fake news detection systems.
  • Ordinal Classification Matters for Nuance: Treating the LIAR dataset's truthfulness labels as ordered categories (e.g., "pants on fire" is worse than "false") improved classification accuracy and shifted the model's behavior towards more precise predictions, especially for extreme categories. This highlights the importance of considering the inherent structure of labels in classification tasks.
  • Feature Importance Reveals Deeper Characteristics: The feature ablation experiments provided valuable insights into which linguistic features are most discriminative for fake news detection. For instance, LIWC features, syntactic complexity, and lexical diversity were found to be particularly important, offering clues about the linguistic patterns that differentiate real from fake news.

Key Results

  • Competitive Accuracy: The BRNN models trained on interpretable features achieved accuracy scores highly competitive with state-of-the-art models. For the ISOT dataset, their BRNN CoCoGen model reached 99.3% accuracy, very close to the best-performing models at 99.8%. For the LIAR dataset, their BRNN CoCoGen (ordered) model with context and speaker profile achieved 27.2% accuracy, outperforming some previous black-box models.
  • Feature Importance Insights: Ablation studies revealed that LIWC features, syntactic complexity, lexical richness, and n-grams were the most influential feature groups for fake news detection on both datasets. Dropping these groups led to significant drops in accuracy, indicating their crucial role.
  • Ordinal Classification Benefits: Applying an ordinal classification approach to the LIAR dataset resulted in a slight increase in accuracy (0.6%) and a more balanced distribution of errors in the confusion matrix, with fewer misclassifications of extreme categories.

Limitations and Future Work

  • Limitations: The paper acknowledges that while their approach is promising, it still relies on complex deep learning models (BRNNs). The "interpretability" comes from the features used, but the internal workings of the BRNN itself are not fully transparent. The performance on the LIAR dataset, while competitive, is still relatively low (around 27%), indicating the inherent difficulty of classifying short, nuanced claims.
  • Future Work: The authors plan to extend their approach to fake news detection in German, as research in this area lags behind English. They also aim to apply their methods to detect rumors and conspiracy theories, particularly in the context of combating misinformation related to events like the COVID-19 pandemic.

Summary updated

Abstract

‘Fake news’ – succinctly defined as false or misleading information masquerading as legitimate news – is a ubiquitous phenomenon and its dissemination weakens the fact-based reporting of the established news industry, making it harder for political actors, authorities, media and citizens to obtain a reliable picture. State-of-the art language-based approaches to fake news detection that reach high classification accuracy typically rely on black box models based on word embeddings. At the same time, there are increasing calls for moving away from black-box models towards white-box (explainable) models for critical industries such as healthcare, finances, military and news industry. In this paper we performed a series of experiments where bi-directional recurrent neural network classification models were trained on interpretable features derived from multi-disciplinary integrated approaches to language. We apply our approach to two benchmark datasets. We demonstrate that our approach is promising as it achieves similar results on these two datasets as the best performing black box models reported in the literature. In a second step we report on ablation experiments geared towards assessing the relative importance of the human-interpretable features in distinguishing fake news from real news.
BibTeX
@inproceedings{qiao2020language,
  author = {Yu Qiao and Daniel Wiechmann and Elma Kerz},
  title = {{A Language-Based Approach to Fake News Detection Through Interpretable Features and BRNN}},
  booktitle = {UvA-DARE},
  year = {2020},
}
Figure