MisinfoBench: A Multi-Dimensional Benchmark for Evaluating LLMs’ Resilience to Misinformation

Ye Yang, Donghe Li, Zuchen Li, Fengyuan Li, Jingyi Liu, Li Sun, Qingyu YangEMNLP 2025

TL;DR

Large Language Models (LLMs) are increasingly susceptible to misinformation, especially in multi-turn conversations. This paper introduces MisinfoBench, a novel benchmark designed to rigorously evaluate LLMs' ability to discern, resist, and reject misinformation across various scenarios. The findings reveal significant performance gaps, particularly between proprietary and open-source models, highlighting the urgent need for more robust LLM defenses against deceptive content.

What Problem Does This Paper Address?

Large Language Models (LLMs) have demonstrated remarkable capabilities across numerous Natural Language Processing (NLP) tasks. However, their widespread deployment in applications like search engines and virtual assistants has exposed a critical vulnerability: susceptibility to misinformation. Existing benchmarks, such as TruthfulQA and FEVER, primarily assess factual accuracy in isolated queries. They fail to capture the nuances of real-world interactions where misinformation can accumulate over multiple turns in a dialogue, or where users might intentionally introduce misleading context or false premises. This gap leaves a critical need for a benchmark that can systematically evaluate LLMs' resilience to misinformation in dynamic, interactive settings, which is precisely what this paper aims to address.

Methods

The authors introduce MisinfoBench, a multi-dimensional benchmark designed to evaluate LLMs' resilience to misinformation. The benchmark is structured around three core dimensions:

  • Discernment: The ability to recognize and explain falsehoods in isolated inputs.
  • Resistance: The capability to maintain factual accuracy when exposed to progressively misleading context in multi-turn dialogues.
  • Principled: The capacity to reject or override fabricated materials when explicitly instructed to answer based on them.

These dimensions are operationalized through seven distinct evaluation tasks. To automate the evaluation process, the authors leverage GPT-4o as an LLM judge, a methodology validated through comparative experiments with human expert assessments.

Datasets

MisinfoBench is constructed from two complementary data sources:

  • MT-Dataset: A large-scale, multi-turn dialogue dataset comprising 4,962 conversations (average 20.52 turns per session) sourced from Stack Overflow and Academia Stack Exchange. This dataset provides natural conversational context.
  • Misinfo-Dataset: A manually curated dataset consisting of 2,000 misinformation-based question-answer pairs. This includes 1,000 misleading question-answer pairs (MQ-MA) and 1,000 normal questions paired with misleading answers (NQ-MA). Each entry is annotated with explicit error points, detailing factual inaccuracies or logical fallacies.

Key Insights and Ideas

  • Multi-turn dialogues are a significant vulnerability: LLMs struggle more in conversational settings where misinformation can build up over time, highlighting the limitations of single-turn evaluation.
  • Proprietary models lead the pack: Closed-source models like GPT-4 and Claude 3.5 Sonnet demonstrate substantially higher resilience to misinformation compared to their open-source counterparts. This suggests proprietary optimization pipelines are more effective at balancing instruction following with factual accuracy.
  • Language matters: Performance varies across languages, with English generally showing better results, likely due to training data prevalence. Mixed-language inputs can disrupt misinformation recognition.
  • Model size isn't everything: While larger models generally perform better, size alone doesn't guarantee robustness. Advanced reasoning, alignment, and architectural choices are crucial.
  • The "Principled" dimension is particularly challenging: LLMs often fail when explicitly instructed to rely on fabricated information, indicating a need for stronger mechanisms to refuse harmful or misleading instructions.

Key Results

The paper evaluates 16 LLMs (13 open-source, 3 proprietary) on MisinfoBench. Key findings include:

  • Proprietary models outperform: GPT-4 and Claude 3.5 Sonnet achieved significantly higher scores across most tasks, especially in Discernment and Resistance. For instance, GPT-4 achieved near-perfect scores in Discernment.
  • Open-source models lag: Many open-source models showed considerable weaknesses, particularly in multi-turn and principled reasoning tasks. For example, some LLaMA models showed very low scores in the Principled dimension (e.g., 0% for LLaMA3.1-8B-Instruct on RAP).
  • Language-specific performance: In the Immunity to Misguidance (IM) task, models generally performed best in English, with notable degradation in bilingual and Chinese settings. Qwen models showed more balanced performance across languages compared to LLaMA.
  • Dialogue length impact: Shorter dialogues showed stronger resistance, but performance declined as context windows expanded, only improving again after a certain length, suggesting a "contextual forgetting" effect.

Limitations and Future Work

The authors acknowledge that with the rapid development of LLMs, new forms of misinformation and reasoning capabilities may emerge that are not fully captured by the current benchmark. Future work will focus on expanding MisinfoBench to include:

  • More languages and domains.
  • More adversarial scenarios.
  • Investigating dynamic misinformation resistance in extended interactions and real-time adaptation strategies.

They envision MisinfoBench as a catalyst for developing more truthful, context-aware, and ethically aligned AI systems.

Summary updated

BibTeX
@inproceedings{yang2025misinfobench,
  author = {Ye Yang and Donghe Li and Zuchen Li and Fengyuan Li and Jingyi Liu and Li Sun and Qingyu Yang},
  title = {{MisinfoBench: A Multi-Dimensional Benchmark for Evaluating LLMs’ Resilience to Misinformation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure