Stance Detection in Fake News A Combined Feature Representation

Bilal Ghanem, Paolo Rosso, Francisco RangelEMNLP 2018

TL;DR

This paper tackles the growing problem of fake news by proposing a novel approach to stance detection. By combining lexical cues, word embeddings, and n-gram features, the authors achieve state-of-the-art results on the Fake News Challenge (FNC-1) dataset, demonstrating the effectiveness of a hybrid feature representation.

What Problem Does This Paper Address?

The internet has become a breeding ground for fake news and misinformation, posing a significant threat to individuals and society. While manual fact-checking efforts exist, they are time-consuming and struggle to keep pace with the rapid spread of false information. This paper addresses the need for automatic fake news detection, specifically focusing on the "stance detection" task. Stance detection, in this context, aims to determine how an article's content relates to its title, classifying it as "Agree," "Disagree," "Discuss," or "Unrelated." This is crucial because understanding the relationship between a claim (title) and its supporting or refuting evidence (article) is a key step in verifying information.

Methods

The authors propose a feature-rich approach to stance detection, moving beyond single-feature methods. Their core methodology involves:

  • Preprocessing: Standard text cleaning steps are applied, including removing URLs, hashtags, user mentions, non-English characters, and special characters to reduce noise and bias.
  • Feature Engineering: This is the heart of their contribution. They combine three main types of features:
  • Cue Words: They leverage pre-defined categories of words (e.g., "Belief," "Denial," "Fake") previously used in rumor stance detection. To expand these, they use Google News word2vec to find similar words for each cue word, creating a richer set of indicators.
  • Google News Word2vec Embeddings: They utilize pre-trained word embeddings to capture semantic meaning. Importantly, they experiment with both full sentence embeddings (averaged word embeddings) and embeddings derived from key sentence components (verbs, nouns, adjectives), finding the latter to be more effective for plagiarism detection in prior work.
  • FNC-1 Baseline Features: They also incorporate features used in the original FNC-1 challenge baseline, which include n-gram co-occurrence between titles and articles, word overlapping, and the presence of polarized words.
  • Classification: They experiment with various classifiers, ultimately finding a Neural Network (NN) with two hidden layers and ReLU activation to perform best.

Datasets

The primary dataset used in this paper is the Fake News Challenge (FNC-1) dataset. This dataset consists of pairs of news titles and articles, labeled with one of four stance categories: Agree, Disagree, Discuss, and Unrelated. The paper notes that the dataset is highly imbalanced, with "Unrelated" being the majority class. The authors also mention that the "Unrelated" class articles were artificially created, which can pose challenges for feature extraction.

Key Insights and Ideas

  • Hybrid Features are Key: The paper strongly suggests that combining different types of features (lexical, semantic, and structural) is more effective than relying on any single feature type for stance detection.
  • Cue Words Matter, but Need Expansion: While cue words provide valuable signals, their effectiveness is significantly boosted by expanding them using word embeddings to capture a broader semantic range.
  • Importance of "Report" and "Negation" Cues: The analysis of cue word importance reveals that "Report" is the most discriminative category, followed by "Negation" and "Belief." This highlights the significance of identifying explicit claims and denials.
  • Dataset Imbalance is a Challenge: The heavily skewed distribution of classes in the FNC-1 dataset, particularly the large "Unrelated" class, presents a significant hurdle for accurate stance detection. The authors acknowledge that the "Unrelated" class's artificial creation might make its features less meaningful.
  • Feature Importance Varies by Class: The study's deeper dive into feature importance per class shows that disagreement-related cue words are more important for "Disagree" classes than for "Agree" classes, which is an intuitive finding.

Key Results

The proposed approach achieved a Macro F1 score of 0.596 on the FNC-1 dataset. This result is very close to the state-of-the-art at the time of publication, with only a 0.013 difference. Individually, the Word2vec embeddings feature set achieved a Macro F1 of 0.488, and the cue words feature set achieved 0.250. The combination of cue words with other features improved the overall result by 2.5%.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges the artificial creation of the "Unrelated" class in the FNC-1 dataset, which might affect the meaningfulness of feature analysis for this class.
  • The significant length difference between titles and articles in the dataset is noted as a challenge.
  • Future Work:
    • The authors propose summarizing the articles in the dataset as a potential avenue to improve stance detection, given the length disparity. This could help in creating more comparable text fragments.

Summary updated

BibTeX
@inproceedings{ghanem2018stance,
  author = {Bilal Ghanem and Paolo Rosso and Francisco Rangel},
  title = {{Stance Detection in Fake News A Combined Feature Representation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure