CoCoLoFa: A Dataset of News Comments with Common Logical Fallacies Written by LLM-Assisted Crowds

Min-Hsuan Yeh, Ruyuan Wan, Ting-Hao Huang

Introduction: The Challenge of Logical Fallacies in Online Discourse

Logical fallacies—rhetorical flaws like the "slippery slope" or "false dilemma"—frequently undermine the quality of online debate. While identifying these errors is essential for promoting healthy discourse and mitigating misinformation, automating their detection remains a significant challenge. The primary obstacle is the lack of high-quality, contextualized data. Existing datasets are often small, lack non-fallacious "negative" examples, or isolate arguments from their original news context, making it difficult for machine learning models to grasp the nuances of real-world reasoning.

Examples of logical fallacies in news comments, such as Slippery Slope and False Dilemma.

Methodology: LLM-Assisted Crowdsourcing

To address the data bottleneck, researchers developed CoCoLoFa, a dataset generated through a novel "human-in-the-loop" strategy. Rather than asking crowd workers to simply identify fallacies in existing text—a task that often yields inconsistent results—the team tasked 143 participants with writing original comments in response to actual news articles.

Recognizing that crafting specific rhetorical errors is a high-difficulty task requiring specialized knowledge, the researchers integrated an LLM-powered assistant (GPT-4) into the worker interface. The workflow followed a structured path: workers first read a news article and passed an attention check; they then drafted a comment attempting to embody a specific fallacy, such as an "Appeal to Tradition." The LLM assistant then provided real-time suggestions and examples, helping workers refine their drafts into subtle, realistic fallacies. The resulting comments were then validated by expert linguists (PhDs) to ensure high quality and accurate labeling.

The CoCoLoFa Dataset

CoCoLoFa represents the largest and most comprehensive resource of its kind. It includes over 7,700 comments linked to 648 news articles, covering eight common fallacy types: Appeal to Authority, Majority, Nature, Tradition, Worse Problems, False Dilemma, Hasty Generalization, and Slippery Slope.

Crucially, the dataset includes over 3,100 negative samples—neutral arguments without fallacies—to provide a robust baseline for classification models. As shown in the table below, CoCoLoFa significantly exceeds previous benchmarks like LOGIC and Argotario in terms of total items, vocabulary size, and the inclusion of negative samples.

Comparison of CoCoLoFa with existing logical fallacy datasets.

Key Insights and Results

The study yielded several critical findings for the field of Natural Language Processing (NLP) and discourse analysis:

  • Small Models, Big Performance: One of the most striking results was that relatively small models (BERT and RoBERTa), when fine-tuned specifically on CoCoLoFa, achieved an F1 score of 0.86 for fallacy detection. This significantly outperformed massive, general-purpose models like GPT-4o in zero-shot and few-shot settings.
  • The Importance of Context: Models that incorporated news article titles and parent comments as context performed better than those analyzing comments in isolation. This confirms that fallacies are rarely standalone errors but are deeply tied to the specific flow of a conversation.
  • The Subjectivity Trap: Even among expert linguists, inter-annotator agreement remained moderate (Cohen’s κ ≈ 0.47). This highlights a fundamental challenge in the field: what one person perceives as a "valid rhetorical strategy," another may see as a "logical fallacy."
  • Generalizability and Real-World Complexity: Models trained on CoCoLoFa showed superior "transfer learning" capabilities, performing better on external datasets (like Reddit) than models trained on those datasets themselves. However, when tested on "in the wild" comments from the New York Times, performance dropped significantly (best F1 of 0.34), highlighting the messiness of organic online arguments compared to controlled environments.

Conclusion and Future Work

CoCoLoFa demonstrates that LLMs are most effective when used as "scaffolding" for human creativity, enabling the creation of high-quality datasets for complex linguistic tasks. By assisting rather than replacing human workers, the researchers captured human-like diversity while maintaining structural integrity. While currently limited to eight fallacy types, future work aims to expand the dataset to include more categories, such as ad hominem attacks, and to develop models capable of providing rationales—explaining why a specific argument is fallacious rather than just labeling it.

Summary updated

Abstract

Detecting logical fallacies in texts can help users spot argument flaws, but automating this detection is not easy. Manually annotating fallacies in large-scale, real-world text data to create datasets for developing and validating detection models is costly. This paper introduces CoCoLoFa, the largest known logical fallacy dataset, containing 7,706 comments for 648 news articles, with each comment labeled for fallacy presence and type. We recruited 143 crowd workers to write comments embodying specific fallacy types (e.g., slippery slope) in response to news articles. Recognizing the complexity of this writing task, we built an LLM-powered assistant into the workers’ interface to aid in drafting and refining their comments. Experts rated the writing quality and labeling validity of CoCoLoFa as high and reliable. BERT-based models fine-tuned using CoCoLoFa achieved the highest fallacy detection (F1=0.86) and classification (F1=0.87) performance on its test set, outperforming the state-of-the-art LLMs. Our work shows that combining crowdsourcing and LLMs enables us to more effectively construct datasets for complex linguistic phenomena that crowd workers find challenging to produce on their own.
BibTeX
@inproceedings{yeh2024cocolofa,
  author = {Min-Hsuan Yeh and Ruyuan Wan and Ting-Hao Huang},
  title = {{CoCoLoFa: A Dataset of News Comments with Common Logical Fallacies Written by LLM-Assisted Crowds}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.39},
}
Figure