ContraQA: Question Answering under Contradicting Contexts

Liangming Pan, Wenhu Chen, Min-Yen Kan, William Yang Wang

Abstract

With a rise in false, inaccurate, and misleading information in propaganda, news, and social media, real-world Question Answering (QA) systems face the challenges of synthesizing and reasoning over contradicting information to derive correct answers. This urgency gives rise to the need to make QA systems robust to misinformation, a topic previously unexplored. We study the risk of misinformation to QA models by investigating the behavior of the QA model under contradicting contexts that are mixed with both real and fake information. We create the first large-scale dataset for this problem, namely Contra-QA, which contains over 10K human-written and model-generated contradicting pairs of contexts. Experiments show that QA models are vulnerable under contradicting contexts brought by misinformation. To defend against such a threat, we build a misinformation-aware QA system as a counter-measure that integrates question answering and misinformation detection in a joint fashion.

The Misinformation Challenge in Open-Domain QA

Current open-domain question answering (ODQA) systems are built on a fragile assumption: that their source material is 100% trustworthy. Most systems follow a "retrieve-then-read" pipeline, searching a massive corpus like Wikipedia for relevant documents and then using a "reader" model to extract an answer. Because researchers typically use Wikipedia as a gold standard, these models haven't been trained to handle contradictions or verify the truth.

However, the real-world web is increasingly "polluted" with misinformation. As large language models (LLMs) make it easier to generate realistic-looking fake news at scale, we must understand how vulnerable our QA systems are to "corpus pollution."

Overview of the misinformation-polluted Wikipedia corpus and the resulting failure in the QA pipeline.

Methods: Simulating Corpus Pollution

To investigate this vulnerability, we developed a framework to simulate misinformation attacks using two primary methods:

  1. Manual Creation: Workers via Amazon Mechanical Turk rewrote Wikipedia passages, making "long edits" that changed key facts (dates, locations, names) while maintaining fluency.
  2. BART-FG (Fake Passage Generator): A novel generative model designed to produce controllable misinformation. It uses Span Masking to identify key entities or linguistic structures and Gap Span Filling (GSF) to fill these masks using global and local context, ensuring the fake information appears authentic within the document.

We "polluted" a clean corpus of 1 million Wikipedia passages with these fake documents and tested five attack modes, ranging from random fact-swapping to "Targeted" attacks that specifically overwrite the answer to a known question.

Key Results: A Universal Vulnerability

Our evaluation of five state-of-the-art models—RoBERTa, SpanBERT, Longformer, ELECTRA, and DeBERTaV3—revealed a universal vulnerability to misinformation. Even when misinformation made up only a small fraction of the corpus, models were frequently misled.

  • The Targeted Attack: When an attacker generates a fake passage to specifically answer a known question incorrectly, the results are devastating. For DeBERTaV3, the Exact Match (EM) score plummeted from 62.30 to 29.52—a relative drop of over 50%.
  • The "Relevance" Trap: Surprisingly, more accurate retrievers (like ColBERT) can be more vulnerable. Because fake passages are engineered to contain the exact keywords of a question, retrievers often rank misinformation higher than the true document.
  • Scalability of AI Misinformation: AI-generated misinformation via BART-FG proved just as effective at fooling models as human-written text, suggesting that malicious actors could automate the poisoning of information sources at an unprecedented scale.
Performance degradation (EM) of QA models as the number of injected fake passages increases across different attack types.

Discussion and Future Directions

The study highlights that current QA models rely on "spurious patterns"—matching words between the question and text—rather than deep comprehension. Because they do not reason about the truth, they cannot resolve contradictions when presented with conflicting "answers."

This research calls for a new generation of "misinformation-aware" QA systems. Future work should focus on integrating fact-checking directly into the QA pipeline, allowing models to identify and discard contradictory or untrustworthy evidence before attempting to generate an answer.

Summary updated

Abstract

With a rise in false, inaccurate, and misleading information in propaganda, news, and social media, real-world Question Answering (QA) systems face the challenges of synthesizing and reasoning over contradicting information to derive correct answers. This urgency gives rise to the need to make QA systems robust to misinformation, a topic previously unexplored. We study the risk of misinformation to QA models by investigating the behavior of the QA model under contradicting contexts that are mixed with both real and fake information. We create the first large-scale dataset for this problem, namely Contra-QA, which contains over 10K human-written and model-generated contradicting pairs of contexts. Experiments show that QA models are vulnerable under contradicting contexts brought by misinformation. To defend against such a threat, we build a misinformation-aware QA system as a counter-measure that integrates question answering and misinformation detection in a joint fashion.
BibTeX
@inproceedings{pan2022contraqa,
  author = {Liangming Pan and Wenhu Chen and Min-Yen Kan and William Yang Wang},
  title = {{ContraQA: Question Answering under Contradicting Contexts}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=Ybx635VOYoM},
}
Figure