Infusing Knowledge from Wikipedia to Enhance Stance Detection

Zihao He, Negar Mokhberian, Kristina LermanACL 2022

TL;DR

This paper introduces WS-BERT, a novel approach that leverages Wikipedia knowledge to significantly improve stance detection performance. By infusing Wikipedia's factual information about targets into BERT-based models, WS-BERT achieves state-of-the-art results across various stance detection tasks, including target-specific, cross-target, and zero/few-shot scenarios.

What Problem Does This Paper Address?

Stance detection, the task of identifying an author's attitude (favor, neutral, against) towards a specific target, is inherently challenging when the model lacks background knowledge about that target. This is particularly true when the text doesn't explicitly mention the target, forcing human readers to rely on their existing knowledge to infer the stance. Existing stance detection models often fail to incorporate this crucial background information, leading to suboptimal performance. The authors aim to bridge this gap by using Wikipedia as a rich source of background knowledge to enhance stance detection models.

Methods

The core contribution is the Wikipedia Stance Detection BERT (WS-BERT) model. The authors propose two variants of WS-BERT to handle different document styles:

  • WS-BERT-Single: For formal documents (like Wikipedia articles themselves), this variant uses a single BERT model to jointly encode the document, target, and Wikipedia knowledge. The input format is structured as [CLS] Text: d Target: t [SEP] w [SEP], allowing all components to attend to each other.
  • WS-BERT-Dual: For informal documents (like social media posts), this variant uses two separate BERT-based language models. One model (e.g., BERTweet or COVID-Twitter-BERT) encodes the document and target, while a separate BERT model encodes the Wikipedia knowledge. The pooled outputs from both models are then concatenated to form the final representation. This dual-model approach minimizes domain shift between the informal document text and the more formal Wikipedia text.

The final representation from either WS-BERT variant is fed into a fully-connected layer with a softmax activation for stance prediction, optimized using cross-entropy loss.

Datasets

The authors evaluate their model on three recently published benchmark datasets:

  • P-Stance: A dataset of tweets related to politicians, used for target-specific and cross-target stance detection.
  • COVID-19-Stance: A dataset of pandemic-related tweets with four targets, used for target-specific stance detection.
  • Varied Stance Topics (VAST): A dataset of comments from "Room for Debate" sections, used for zero-shot and few-shot stance detection.

For all datasets, the authors used summaries of fetched Wikipedia pages as the textual background knowledge. For VAST, they used an API to crawl Wikipedia pages, and for targets without Wikipedia pages, they used the target text itself.

Key Insights and Ideas

  • Wikipedia as a valuable knowledge source: The paper strongly advocates for using Wikipedia's factual information as a readily accessible and effective source of background knowledge for stance detection.
  • Handling domain shift is crucial: The distinction between WS-BERT-Single and WS-BERT-Dual highlights the importance of adapting the encoding strategy based on the formality of the input documents to mitigate domain shift issues.
  • Background knowledge is vital even with ample data: Even for target-specific stance detection, where models are trained on substantial data for each target, infusing Wikipedia knowledge leads to significant performance improvements. This suggests that explicit factual knowledge complements learned patterns.
  • Zero-shot/Few-shot learning benefits most: The performance gains are most pronounced in zero-shot and few-shot scenarios, where models have limited or no direct training examples for a target. This underscores the critical role of background knowledge when learning is data-scarce.
  • Robustness to noisy data: The authors note that even with potentially noisy Wikipedia summaries obtained via automated crawling, their method still significantly outperforms baselines, demonstrating robustness.

Key Results

WS-BERT consistently achieves state-of-the-art results across all evaluated datasets and sub-tasks:

  • Target-specific stance detection: WS-BERT-Dual significantly outperforms strong baselines like BERTweet and COVID-Twitter-BERT on both P-Stance and COVID-19-Stance datasets, achieving improvements of up to 2.3% and 2.4% in macro-average F1 scores, respectively.
  • Cross-target stance detection: On P-Stance, WS-BERT-Dual achieves new state-of-the-art results, outperforming BERTweet by a notable margin, especially for target pairs where transfer learning is more challenging (e.g., "Trump" to "Biden").
  • Zero-shot and Few-shot stance detection: WS-BERT-Single significantly improves performance on the VAST dataset, creating a new state-of-the-art with a substantial margin (e.g., 4.5% improvement in overall F1 score over the best baseline).

Limitations and Future Work

  • Manual Wikipedia page selection: For P-Stance and COVID-19-Stance, the Wikipedia pages were manually fetched, which is not scalable. While an API was used for VAST, the retrieved pages might not always be the most relevant.
  • Potential for noisy information: Automated retrieval of Wikipedia summaries can introduce noise or miss crucial content. However, the paper demonstrates the model's robustness to such noise.
  • Future Work:
    • Investigate incorporating knowledge about entities within the input documents, not just about the targets.
  • Address the potential bias present in Wikipedia articles and how to mitigate its negative impact on stance modeling.
  • Explore the utility of background knowledge from relevant news articles.

Summary updated

BibTeX
@inproceedings{he2022infusing,
  author = {Zihao He and Negar Mokhberian and Kristina Lerman},
  title = {{Infusing Knowledge from Wikipedia to Enhance Stance Detection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure