Belittling the Source: Trustworthiness Indicators to Obfuscate Fake News on the Web

Diego Esteves, Aniketh Janardhan Reddy, Piyush Chawla, Jens LehmannEMNLP 2018

TL;DR

This paper tackles the growing problem of fake news by proposing an automated system to assess website credibility. By extracting novel "bag-of-tags" features from HTML and combining them with textual analysis, they achieve state-of-the-art results in classifying websites as credible or not, offering a practical solution to filter misinformation.

What Problem Does This Paper Address?

The internet has become a breeding ground for misinformation, with fake news spreading rapidly and having serious consequences. While fact-checking tools are essential, a crucial step is determining the credibility of the information source itself. Existing methods for assessing website credibility face several challenges: manual annotation is costly and time-consuming, publicly available web indicators like Google PageRank are shut down or not free, and existing databases are often manually curated and don't scale. Furthermore, much of the research is theoretical or relies on private, inaccessible data. This paper aims to bridge this gap by developing an automated model that can assign a credibility score to websites, helping to identify trustworthy sources and flag dubious ones.

Methods

The authors propose a novel approach that moves beyond traditional social-based features (like social media shares or PageRank) which can be costly and prone to manipulation. Instead, they focus on content-based features and introduce a new concept called "bag-of-tags."

  • Bag-of-Tags: This technique treats HTML tags similarly to words in a document. They group HTML tag occurrences and then encode them into a sequence, allowing for document classification based on this structural information. This approach is language-agnostic and can be computed from readily available website data.
  • Textual Features: They also leverage linguistic and textual features, including spelling errors, text category classification (using Naive Bayes, LexRank, and LSA), readability metrics, and spam detection.
  • Feature Selection: To optimize performance, they employ feature selection techniques like ANOVA SelectKBest.
  • Model Training: They train classifiers (like Gradient Boosting and AdaBoost) on these features to predict credibility scores.

Datasets

The paper utilizes two primary datasets for training and evaluation:

  • Microsoft Dataset: This dataset contains thousands of URLs with five-point Likert scale credibility ratings, ranging from 1 (very non-credible) to 5 (very credible). The definition of credibility used is: "A credible webpage is one whose information one can accept as the truth without needing to look elsewhere."
  • Content Credibility Corpus (C3): This is a more recent and larger dataset, containing over 15,000 evaluations from over 2,000 participants, with additional website characteristics and user demographics. The authors specifically extracted the URLs and their corresponding five-point Likert scale ratings from this corpus to align with the Microsoft dataset.

Additionally, they used the FactBench dataset for a real-world use-case evaluation, where they collected URLs that provided evidence for claims and then assessed their credibility using their model.

Key Insights and Ideas

  • Bag-of-Tags are Powerful: The "bag-of-tags" approach, which analyzes HTML structure, proves to be a surprisingly effective feature for predicting website credibility, even outperforming some purely textual methods. This highlights the importance of structural cues in assessing trustworthiness.
  • Social Features are Problematic: The authors argue against relying heavily on social-based features (like PageRank, Facebook shares) due to their financial costs, reliance on third-party APIs, and susceptibility to manipulation. This is a significant departure from some prior work.
  • Visual Features Underestimated: While their model doesn't explicitly use visual features, they note that research consistently shows visual aspects are important, and it's surprising that some studies (and their own initial feature selection) downplay their significance.
  • "High" Credibility is Easier to Detect: The model is generally good at identifying highly credible sources (F1 scores around 0.80), but detecting "low" credibility sources remains a challenge, with recall being an issue.
  • Reproducibility Challenges: The paper highlights significant reproducibility issues with existing datasets and research, particularly the Microsoft dataset, which they address by releasing a fixed version and cached websites for C3.

Key Results

  • The proposed model, combining textual and "bag-of-tags" features, achieved state-of-the-art results in the 2-class and 5-class credibility classification settings.
  • In the 2-class setting on the Microsoft dataset, their model achieved an average F1 score of 0.772, an improvement of nearly 3% over previous methods.
  • For the C3 Corpus, their model achieved an F1 score of 0.674 for the 2-class setting.
  • In a real fact-checking scenario, their model correctly identified around 80% of URLs associated with positive claims and 70% of non-credible websites linked to false claims.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that detecting "low" credibility sources is still an issue, with recall being a challenge.
  • While they addressed reproducibility issues with existing datasets, the inherent complexities of web data (e.g., dynamic content) can still pose challenges.
  • The "bag-of-tags" approach, while effective, might not capture all nuances of website design and user experience, which are known to influence credibility.
  • Future Work:
    • The authors plan to explore deep learning methods for the HTML2Seq module to further enhance the model's capabilities.
  • They suggest that reinforcement learning methods could be a promising avenue for adapting to the "cat-and-mouse play" between credibility evaluators and those attempting to manipulate models.

Summary updated

BibTeX
@inproceedings{esteves2018belittling,
  author = {Diego Esteves and Aniketh Janardhan Reddy and Piyush Chawla and Jens Lehmann},
  title = {{Belittling the Source: Trustworthiness Indicators to Obfuscate Fake News on the Web}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure