Abstract
Logical fallacy detection is a critical component of computational argumentation and critical thinking, yet it remains a challenging task due to fragmented taxonomies and the inherent subjectivity of human reasoning. This paper introduces MAFALDA (Multi-level Annotated Fallacy Dataset), a unified benchmark designed to bring rigor to the field. By implementing a multi-level taxonomy and a novel "disjunctive" annotation scheme, the researchers provide a framework that embraces multiple valid interpretations of flawed arguments. Our evaluation of state-of-the-art Large Language Models (LLMs) shows that while models like GPT-3.5 lead the field, a significant performance gap remains between AI and human experts, particularly in fine-grained classification.
Introduction
Detecting fallacies—such as the "Straw Man" or "False Dilemma"—is essential for navigating modern discourse, yet it has historically been a "dark art" in Natural Language Processing (NLP). Previous research has suffered from three main issues: inconsistent naming conventions, narrow dataset domains (e.g., exclusively political speeches), and a rigid "single correct answer" approach that ignores the subjective nature of identifying fallacious reasoning. MAFALDA addresses these gaps by unifying existing datasets and allowing for nuanced, multi-label interpretations.
The MAFALDA Framework
Multi-Level Taxonomy
The researchers organized fallacies into a hierarchical structure to allow for evaluation at different levels of granularity:
- Level 0: A binary classification (Fallacy vs. No Fallacy).
- Level 1: Categorization based on Aristotelian pillars: Fallacy of Logic, Fallacy of Credibility, or Appeal to Emotion.
- Level 2: 23 fine-grained fallacy types, such as Ad Hominem, Slippery Slope, and False Causality.
Disjunctive Annotation and Metrics
Unlike traditional benchmarks that force a single label, MAFALDA uses a disjunctive annotation scheme. This acknowledges that a single text span may legitimately be interpreted as multiple types of fallacies. If a model selects any label within the set of human-validated interpretations, it is considered correct. To support this, the authors adapted standard Precision and Recall metrics to handle sets of labels, ensuring models are not unfairly penalized for valid alternative interpretations.
Dataset Composition
The MAFALDA corpus consists of 9,745 English-language texts sourced from Reddit, news outlets, and American political debates. To ensure high quality, the authors created a "Gold Standard" subset of 200 texts (containing 260 fallacy instances) which were manually re-annotated from scratch. Each gold-standard entry includes a manual justification and a breakdown of the argument's formal structure, requiring approximately 12 minutes of expert labor per example.
Results and Analysis
The study evaluated 12 open-source models (including LLaMA2, Mistral, and Vicuna) and GPT-3.5 in a zero-shot setting, comparing them against human performance.
Key Findings
- Model Performance: GPT-3.5 (175B) emerged as the top-performing model across all levels, significantly outperforming open-source alternatives. However, even the best models struggle with the "Appeal to Emotion" category, likely because distinguishing between a persuasive emotional appeal and a fallacious one requires deep contextual understanding.
- The Human-AI Gap: Humans still significantly outperform AI. On the gold standard sample, humans achieved an F1 score of 0.749 at Level 0, compared to GPT-3.5’s 0.627.
- Granularity Challenges: As the task becomes more specific, performance drops sharply for both humans and machines. At Level 2 (fine-grained classification), humans averaged an F1 of 0.186, while GPT-3.5 managed only 0.138, highlighting the extreme difficulty of nuanced fallacy identification.
Conclusion
MAFALDA demonstrates that subjectivity in fallacy detection is a feature, not a bug. By providing a benchmark that accounts for legitimately differing opinions, the researchers have created a more realistic testing ground for critical thinking in AI. Future work will focus on expanding the gold-standard dataset and exploring "Chain-of-Thought" prompting to help models better replicate the complex reasoning required to identify sophisticated logical errors.
Summary updated