Zero-shot stance detection: Paradigms and challenges

Emily Allaway, Kathleen R. McKeown

TL;DR

This paper tackles the challenge of zero-shot stance detection (ZSSD), aiming to build models that can understand opinions on new topics without prior examples. The authors highlight the limitations of current English-centric ZSSD and advocate for expanding research to multilingual and multi-genre settings, proposing best practices for more robust evaluation.

What Problem Does This Paper Address?

Stance detection, the task of identifying an author's opinion (e.g., for, against, neutral) towards a specific target or topic, faces a significant hurdle: the sheer, ever-growing diversity of topics. It's impractical to collect and label data for every possible stance-taking scenario. This is further complicated by the fact that opinions are expressed across numerous languages and writing styles (genres).

Traditional stance detection models often require labeled data for the specific topics they need to analyze. To overcome this, "zero-shot stance detection" (ZSSD) aims to enable models to generalize to topics they haven't seen during training. While ZSSD has gained traction in English, the authors argue that this focus is too narrow. They contend that for stance detection to be truly useful in the real world, it must also handle unseen languages and genres without explicit training examples for them.

Methods

The paper discusses two main paradigms for ZSSD in English:

  • MANY-TOPIC (ZSSD Topic): Models are trained on a large number of topics but with very few examples per topic. Evaluation is on topics completely unseen during training.
  • FEW-TOPIC (ZSSD Topic): Models are trained on a few topics with many examples. Evaluation is again on unseen topics.

The authors survey existing methods for ZSSD Topic, categorizing them into:

  • Learning latent topics: Discovering underlying thematic structures to help generalize to new topics.
  • Leveraging pragmatic information: Identifying key words and phrases that signal stance, regardless of the specific topic.
  • Learning topic-invariant features: Developing representations that are independent of the specific topic, making them transferable. This is often achieved through contrastive learning (making similar examples closer and dissimilar ones further apart) or adversarial learning (training a model to be good at stance detection while being bad at predicting the topic).
  • Incorporating external knowledge: Using external sources like Wikipedia or commonsense knowledge bases to enrich understanding.

Datasets

The paper focuses on two primary datasets for English ZSSD Topic:

  • VAST (Value Alignment Stance Taking): A large dataset with over 4,600 topics, designed for the MANY-TOPIC setting.
  • Sem16: A dataset of English tweets on six topics, adapted for the FEW-TOPIC setting by treating each topic as a zero-shot test set.

The authors also mention other datasets used for cross-topic stance detection and discuss the limitations of existing multilingual and multi-genre datasets, noting that they are often not designed for true zero-shot evaluation.

Key Insights and Ideas

  • The limitations of English-centric ZSSD: The paper strongly advocates for expanding ZSSD research beyond English to encompass multilingual and multi-genre scenarios, arguing that current work is insufficient for real-world applicability.
  • External knowledge is a powerful tool: Incorporating information from external sources like Wikipedia significantly boosts performance in ZSSD, especially for MANY-TOPIC settings.
  • Topic-invariant features are crucial for generalization: Techniques like contrastive and adversarial learning that aim to create topic-agnostic representations are key to enabling models to handle unseen topics.
  • Cross-topic stance is not true ZSSD: While related, cross-topic stance detection often assumes semantic similarity between training and test topics, which may not hold in broader zero-shot scenarios.
  • Robustness and explainability are critical: Simply achieving high overall accuracy isn't enough. Models need to be robust to linguistic nuances like sarcasm and be explainable to build trust and ensure real-world viability.

Key Results

  • For MANY-TOPIC ZSSD on VAST, incorporating external knowledge (especially from Wikipedia) yielded the best results. Models that combined topic-invariant features with other techniques also showed strong performance.
  • In FEW-TOPIC ZSSD on Sem16, methods using external knowledge and contrastive learning were successful. Notably, models using external knowledge performed better on concrete topics (like individuals) than abstract ones.
  • Adversarial learning showed significantly better performance in the FEW-TOPIC setting compared to MANY-TOPIC.
  • The paper highlights that models incorporating general knowledge and features are more adaptable to evolving discourse spaces than those relying on fixed latent topics or pragmatics derived solely from training data.

Limitations and Future Work

The authors identify several limitations and areas for future work:

  • Expansion to Multilingual and Multi-Genre ZSSD: This is the paper's central call to action. Current work in these areas is limited, and datasets are often small or not designed for true zero-shot evaluation.
  • Systematizing Evaluation: The paper proposes best practices for ZSSD, including the use of zero-shot development sets for hyperparameter tuning, explicit control for domain overlap in multi-dataset learning, and comprehensive evaluation across all combinations of zero-shot aspects (e.g., topic, language, genre).
  • Robustness and Explainability: Future work should focus on evaluating models not just on overall accuracy but also on their resilience to adversarial attacks, linguistic challenges (like sarcasm), and their ability to provide interpretable justifications for their predictions.
  • Multi-dataset Learning: While promising, multi-dataset learning for stance detection still faces performance gaps compared to fully-supervised models and requires careful evaluation to ensure true zero-shot transfer.

Summary updated

Abstract

A major challenge in stance detection is the large (potentially infinite) and diverse set of stance topics. Collecting data for such a set is unrealistic due to both the expense of annotation and the continuous creation of new real-world topics (e.g., a new politician runs for office). Furthermore, stancetaking occurs in a wide range of languages and genres (e.g., Twitter, news articles). While zero-shot stance detection in English, where evaluation is on topics not seen during training, has received increasing attention, we argue that this attention should be expanded to multilingual and multi-genre settings. We discuss two paradigms for English zero-shot stance detection evaluation, as well as recent work in this area. We then discuss recent work on multilingual and multi-genre stance detection, which has focused primarily on non-zero-shot settings. We argue that this work should be expanded to multilingual and multi-genre zero-shot stance detection and propose best practices to systematize and stimulate future work in this direction. While domain adaptation techniques are well-suited for work in these settings, we argue that increased care should be taken to improve model explainability and to conduct robust evaluations, considering not only empirical generalization ability but also the understanding of complex language and inferences.
BibTeX
@article{allaway2022zero,
  author = {Emily Allaway and Kathleen R. McKeown},
  title = {{Zero-shot stance detection: Paradigms and challenges}},
  journal = {Frontiers Artif. Intell.},
  year = {2022},
  doi = {10.3389/frai.2022.1070429},
}
Figure