A Deep Ensemble Framework for Fake News Detection and Classification

Arjun Roy, Kingshuk Basak, Asif Ekbal, Pushpak BhattacharyyaarXiv 2018 · arXiv

TL;DR

Researchers from IIT Patna have developed a deep learning ensemble framework that moves beyond simple "true vs. false" binary classification to detect fake news across six nuanced categories. By combining Convolutional Neural Networks (CNNs) and Bi-directional LSTMs (Bi-LSTMs) to process both news statements and speaker metadata, their model achieved a new state-of-the-art accuracy of 44.87% on the benchmark LIAR dataset.

What Problem Does This Paper Address?

Fake news isn't always a binary "black or white" issue; it often exists in shades of gray—exaggerations, half-truths, or statements taken out of context. While most automated detection systems focus on binary classification, this paper tackles fine-grained classification.

The challenge is twofold:

  1. Short Text Constraints: News statements are often very short, providing little linguistic context for a machine (or even a human) to judge authenticity.
  2. Metadata Integration: A speaker's history, political affiliation, and the context of the statement (e.g., a radio ad vs. a committee hearing) are vital clues that are difficult to model effectively alongside raw text.

Methods

The authors propose an ensemble architecture that leverages the strengths of two different neural network types:

  • Bi-LSTM (Bi-directional Long Short Term Memory): Used to capture the sequential nature of the text. By processing sequences both forward and backward, the model understands the temporal dependencies within a statement.
  • CNN (Convolutional Neural Network): Used to extract "hidden features" and local patterns within the text and metadata that might not be apparent through sequential analysis alone.
  • Metadata Modeling: The researchers didn't just dump metadata into the model. They identified 10 specific relations (e.g., "Speaker and Party," "Statement and Context") and fed these into separate network layers.
  • Feature Engineering: They introduced a special feature vector based on a speaker's "credit history"—a count of how many times they have previously been caught in lies or half-truths.
  • The Ensemble: Representations from the CNN and Bi-LSTM branches are merged and passed through a Multi-layer Perceptron (MLP) with a Softmax classifier for the final six-way decision.

Datasets

The study utilizes the LIAR dataset (Wang, 2017), which consists of approximately 12.8K annotated short political statements collected from Politifact.com between 2007 and 2016.

Statements are labeled with one of six classes:

  1. Pants-fire (Blatant lie)
  2. False
  3. Barely-true
  4. Half-true
  5. Mostly-true
  6. True

Each entry includes 11 metadata attributes, including the speaker's job, state, party affiliation, and historical truthfulness counts.

Key Insights and Ideas

  • Metadata is a Force Multiplier: The model's performance improved significantly by channelizing speaker profile attributes through different neural layers rather than treating them as simple flat features.
  • The "True" Confidence Gap: An interesting finding was that while the model had low recall for the "True" class (it missed many true statements), it had very high precision (0.85). This means if the model labels something as "True," it is highly likely to actually be true, even if it's conservative in making that call.
  • Semantic Overlap: Error analysis showed that the model most frequently confuses adjacent categories (e.g., "Pants-fire" with "False," or "True" with "Mostly-true"). This suggests that the linguistic boundaries between these nuanced categories are naturally fuzzy.

Key Results

  • State-of-the-Art Performance: The combined RNN-CNN ensemble achieved an overall accuracy of 44.87%, outperforming the previous state-of-the-art hybrid LSTM model (41.5%) and the original LIAR paper's CNN benchmark (27.4%).
  • Component Contribution: The CNN-only model (42.89%) slightly outperformed the Bi-LSTM-only model (42.65%), but the ensemble of both provided the best results, proving that sequential and local feature extraction are complementary.

Limitations and Future Work

  • The Cold Start Problem: The model struggles with "new" speakers who were not present in the training set or who have very little "credit history" data.
  • Data Scarcity: The authors note that 12.8K instances, while significant, may not be enough to perfectly distinguish between six highly overlapping classes. They suggest that semi-supervised or active learning could help bridge this gap.
  • Sarcasm Detection: The authors hypothesize that fake news detection could benefit from techniques used in sarcasm detection, as both involve identifying a disconnect between literal meaning and intended truth.

Summary updated

Abstract

Fake news, rumor, incorrect information, and misinformation detection are nowadays crucial issues as these might have serious consequences for our social fabrics. The rate of such information is increasing rapidly due to the availability of enormous web information sources including social media feeds, news blogs, online newspapers etc. In this paper, we develop various deep learning models for detecting fake news and classifying them into the pre-defined fine-grained categories. At first, we develop models based on Convolutional Neural Network (CNN) and Bi-directional Long Short Term Memory (Bi-LSTM) networks. The representations obtained from these two models are fed into a Multi-layer Perceptron Model (MLP) for the final classification. Our experiments on a benchmark dataset show promising results with an overall accuracy of 44.87\%, which outperforms the current state of the art.
BibTeX
@misc{roy2018deep,
  author = {Arjun Roy and Kingshuk Basak and Asif Ekbal and Pushpak Bhattacharyya},
  title = {{A Deep Ensemble Framework for Fake News Detection and Classification}},
  year = {2018},
  eprint = {1811.04670},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.04670},
}

Loading related content...

Figure