Paying Attention to Deflections: Mining Pragmatic Nuances for Whataboutism Detection in Online Discourse

Khiem Phi, Noushin Salek Faramarzi, Chenlu Wang, Ritwik BanerjeeACL 2024 · DOI

TL;DR

Researchers have developed a new method called MINA (Mining Negatives with Attention) to detect "whataboutism"—the practice of deflecting criticism by raising a different issue—in online comments. By using cross-attention to identify pragmatically dissimilar examples within a conversation, the model achieves up to a 10% improvement in detection accuracy over previous state-of-the-art methods.

What Problem Does This Paper Address?

Whataboutism is a common logical fallacy (a variant of tu quoque) used to derail arguments and spread misinformation. While it is easy for humans to spot the phrase "What about...", it is incredibly difficult for AI to distinguish between propagandistic deflection and valid argumentation.

For example, asking "What about the environmental impact?" in a policy debate is a legitimate challenge. However, responding to a report on human rights abuses in one country by saying "What about the crimes of [other country]?" is often a tactic to avoid accountability. Current NLP models struggle with this "pragmatic flexibility" because the syntax and semantics of both sentences are nearly identical. The challenge lies in understanding the intent and the discursive context rather than just the words used.

Conceptual framework for whataboutism detection, highlighting the role of discourse context and pragmatic flexibility.

Methods

The authors propose MINA (Mining Negatives with Attention). The core idea is that to understand whataboutism, a model needs to see how a comment relates to the "Question Under Discussion" (QUD) in its specific social context.

  1. Contextual Tuples: Instead of looking at a single comment, the model is trained on tuples containing the target comment plus other comments from the same video or thread.
  2. Cross-Attention for Negative Mining: Standard models often use "cosine similarity" to find similar or different examples, but the authors prove mathematically that in high-dimensional spaces, most vectors are nearly orthogonal (the "near-orthogonality" problem), making cosine similarity a blunt tool.
  3. The MINA Approach: MINA uses a Transformer encoder’s final layer attention weights to measure "pragmatic contrast." It identifies the most pragmatically dissimilar examples in the surrounding discourse to use as "hard negatives" during training. This forces the model to learn the subtle nuances that separate a topical redirection from a bad-faith deflection.
The MINA architecture, utilizing cross-attention to identify hard negatives for training.

Datasets

The study introduces the $TQ^+$ collections, which are significantly larger and more nuanced than previous propaganda datasets:

  • $TQ^+_{YT}$ (YouTube): 1,642 labeled comments from 17 videos across six divisive topics (e.g., the Ukraine war, Roe v. Wade).
  • $TQ^+_{TW}$ (Twitter/X): 1,202 tweet-reply pairs across eight topics.
  • Annotation: Each instance was labeled by three native English speakers who reviewed the entire video or thread context before deciding if a comment constituted whataboutism.

Key Insights and Ideas

  • Pragmatics > Semantics: You cannot detect whataboutism by looking at a sentence in isolation. The relationship between a comment and its neighbors (the pragmatic context) is the key signal.
  • LLMs are Underwhelming: Surprisingly, modern Large Language Models (LLMs) like Llama-2 and GPT-3.5-turbo struggled with this task. While they recognized "canonical" whataboutism, they failed to catch nuanced, rephrased, or sarcastic instances that MINA handled well.
  • The "What About" Trap: The authors found that many "what about" phrases are actually used for "centering" or "accommodation" (valid discourse moves) rather than propaganda. Models that rely too heavily on lexical cues suffer from high false-positive rates.

Key Results

The MINA-enhanced models showed significant performance gains over standard transfer-learning baselines:

  • YouTube: MINA improved the $F_1$ score by 10% over the next best result (increasing SBERT's $F_1$ from 0.07 to 0.58 in the target class).
  • Twitter: The model achieved a 4% improvement over previous state-of-the-art methods.
  • Separability: t-SNE visualizations showed that MINA created much clearer clusters between whataboutism and non-whataboutism compared to standard language models.

Limitations and Future Work

  • Class Imbalance: Like real-world social media, the datasets are imbalanced, which makes training difficult.
  • Randomized Batches: MINA relies on randomized tuple batches, which can lead to slight variance in model performance across different training runs.
  • Subjectivity: The authors acknowledge that labels can be influenced by the annotator's own perspective, suggesting that future work should incorporate "perspectivist" research.
  • LLM Integration: A promising future direction is exploring how to better prompt or fine-tune LLMs using the pragmatic contrast techniques introduced by MINA.

Summary updated

BibTeX
@inproceedings{phi2024paying,
  author = {Khiem Phi and Noushin Salek Faramarzi and Chenlu Wang and Ritwik Banerjee},
  title = {{Paying Attention to Deflections: Mining Pragmatic Nuances for Whataboutism Detection in Online Discourse}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.750},
}
Figure