Abstract
The CLEF-2024 CheckThat! Lab Task 1 addresses the "firehose" problem of misinformation by developing automated systems to identify which claims are worth a fact-checker's time. This year’s competition highlights that while Large Language Models (LLMs) are entering the fray, fine-tuned Transformer models remain the gold standard for detecting check-worthy content across multiple languages and genres. This paper summarizes the methodologies, datasets, and key findings from the 37 participating teams.
Introduction
Fact-checking is a resource-intensive process that cannot keep pace with the sheer volume of content generated on social media and in political discourse. To alleviate this burden, the "check-worthiness estimation" task acts as a critical triage step. It asks: Does this text contain a verifiable factual claim, and is it potentially harmful if left unverified?
The challenge lies in the diversity of the content. A system must distinguish between a harmless opinion, a verifiable fact, and a dangerous piece of misinformation across different "genres" (like tweets vs. formal political debates) and multiple languages (Arabic, Dutch, and English).
Methods
The participating teams employed a variety of Natural Language Processing (NLP) techniques, which can be categorized into three main approaches:
- Pre-trained Language Models (PLMs): This was the most popular approach. Teams fine-tuned models like BERT, RoBERTa, and XLM-RoBERTa on the provided datasets. These models were often enhanced with specific strategies like ensemble methods.
- Large Language Models (LLMs): Several teams experimented with state-of-the-art models including LLaMA 3, Mistral, and GPT-4. These were used either through few-shot prompting or fine-tuning.
- Novel Architectural Tweaks:
- Adapter Fusion: Team Fraunhofer SIT used a resource-efficient approach combining a task-specific adapter with a Named Entity Recognition (NER) adapter.
- Data Pruning: Team FactFinders used ChatGPT to generate prompts for pruning their training data by 44%, removing uninformative sentences to improve model efficiency.
Datasets
The lab provided a multigenre, multilingual corpus. A key feature of this year's task was the inclusion of "current event" data to test model robustness:
- Arabic: Tweets related to the war on Gaza (starting October 2023).
- Dutch: Messages concerning climate change and associated debates (2021–2022).
- English: Transcribed sentences from the US Presidential election debates.
- Spanish: Provided as extra training data, consisting of transcriptions from Spanish politicians.
The data was manually annotated by professionals and experts, following a scheme that considers whether a claim is verifiable and potentially harmful.
Key Results and Insights
The competition saw 37 teams submitting 236 different system runs. Performance was measured using the F1-measure on the "check-worthy" class:
| Language | Top Team | Top F1 Score |
|---|---|---|
| English | FactFinders | 0.802 |
| Dutch | TurQUaz | 0.732 |
| Arabic | IAI Group | 0.569 |
Key Findings:
- The "English Advantage" Persists: Systems consistently performed better on English data compared to Dutch and Arabic, suggesting that disparity in available pre-training data remains a hurdle.
- LLMs are not (yet) a Silver Bullet: While LLMs like GPT-4 showed promise, smaller, specialized Transformers (like RoBERTa-Large) fine-tuned on the specific task often outperformed the general-purpose giants.
- Context and Preprocessing Matter: Top teams utilized sophisticated preprocessing (URL removal, spelling correction) and feature extraction (POS tagging, dependency parsing) to improve signal quality.
Limitations and Future Work
Several areas for future growth were identified:
- Multimodality: Current systems focus almost entirely on text, but misinformation often spreads via images (memes) and video.
- Domain Shift: Models trained on political debates may struggle with medical misinformation or natural disaster reporting.
- Adversarial Robustness: As fact-checking AI improves, those spreading misinformation may use AI to generate "adversarial examples" designed to bypass these filters.
Summary updated