Cross-lingual Evidence Improves Monolingual Fake News Detection

Daryna Dementieva, Alexander PanchenkoACL 2021

TL;DR

This paper introduces a novel approach to fake news detection that leverages "cross-lingual evidence" – essentially, checking if a story's claims are corroborated or refuted across different languages and news sources. By analyzing how information spreads globally, the authors demonstrate a significant improvement in identifying fake news, even outperforming state-of-the-art models.

What Problem Does This Paper Address?

The internet has become a breeding ground for misinformation, with fake news spreading at an alarming rate and causing real-world harm. While many automated fake news detection systems exist, they often operate in a monolingual bubble, focusing solely on the language of the original article. This limitation overlooks a crucial aspect of how information, both true and false, propagates: it crosses linguistic boundaries. The authors hypothesize that genuine news tends to be reported consistently across various languages and media outlets, while fake news might struggle to gain traction or be corroborated internationally. This paper aims to bridge this gap by exploring how cross-lingual information can enhance fake news detection.

Methods

The core of the proposed method, termed "Cross-lingual Evidence (CE)," involves a five-step pipeline:

  1. Text Extraction: The title and content of a news article are extracted.
  2. Text Translation: The article's title is translated into multiple target languages.
  3. Cross-lingual News Retrieval: Search queries are generated using the translated titles to retrieve relevant articles from search engines in those languages.
  4. Cross-lingual Evidence Impact Computation: The top-N retrieved articles are analyzed to assess the authenticity of the original news. This involves comparing the information in the original article with the retrieved articles to determine how many confirm or refute the claims.
  5. News Classification: Based on the evidence gathered in the previous step, the system classifies the news as either fake or true. The hypothesis is that if a majority of cross-lingual results support the original news, it's more likely to be true; contradictions suggest it might be fake.

The CE feature itself is constructed from two components:

  • Content Similarity (Sim): Cosine similarity between sentence embeddings (averaged from M-BERT) of the original and retrieved articles.
  • Source Credibility (AlexaRank): The AlexaRank score of the source of the retrieved news, indicating its general web traffic and perceived credibility.

Datasets

The authors evaluated their approach on three multi-domain datasets:

  • FakeNewsAMT: Contains 240 fake and 240 legitimate articles across six topics (sports, business, entertainment, politics, technology, education).
  • CelebrityDataset: Features 250 fake and 250 legitimate articles focused on rumors and fake reports about celebrities.
  • ReCOVery: A COVID-19 fake news dataset with 665 fake and 1364 true news articles.

They also conducted a manual experiment on a smaller, custom dataset comprising 10 fake and 10 true news articles from various fields (celebrities, science, politics, culture, world) to test the hypothesis in "ideal conditions."

Key Insights and Ideas

  • The "Language Barrier" as a Detector: The core idea that fake news might not translate well or gain consistent traction across languages is a powerful insight. It leverages the inherent diversity of global media to act as a sanity check.
  • Human Intuition Matches the Method: The manual experiment showed that human annotators could effectively distinguish fake news using cross-lingual evidence, mirroring the automated approach. This validates the underlying principle.
  • Monolingual vs. Cross-lingual: The results clearly indicate that relying solely on monolingual evidence is significantly less effective than incorporating cross-lingual signals.
  • Explainability Potential: While deep learning models can be black boxes, the CE feature offers a degree of interpretability. Users can see which languages and how many sources support or refute a claim, providing a more transparent basis for classification.

Key Results

  • Significant Improvements: The proposed CE feature, when combined with strong baselines like BERT and RoBERTa, achieved state-of-the-art (SOTA) results on all tested datasets.
  • Manual Verification Accuracy: Human annotators achieved an average accuracy of 0.95 in distinguishing fake news using cross-lingual evidence.
  • CE Feature Performance: The CE feature by itself outperformed many baseline models on the FakeNewsAMT dataset and was better than some linguistic features on the Celebrity dataset.
  • Monolingual Evidence Underperforms: Using only monolingual English evidence yielded worse results than cross-lingual evidence.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Limited Language Scope: The current study focused on five languages. Expanding to a wider range of languages could further enhance detection capabilities.
  • English-Centric Original News: The experiments primarily used English as the original language. Future work should explore analogous experiments for news originating in other languages.
  • Global News Distribution Bias: The paper notes that the distribution of news globally is not uniform (e.g., US news is more covered in Europe than vice-versa). Accounting for these biases in cross-lingual analysis could be beneficial.
  • Interpretability of Linguistic Features: While CE offers explainability, combining it with linguistic features from transformer models, which are less interpretable, presents an interesting trade-off between performance and explainability.

Summary updated

BibTeX
@inproceedings{dementieva2021cross,
  author = {Daryna Dementieva and Alexander Panchenko},
  title = {{Cross-lingual Evidence Improves Monolingual Fake News Detection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure