Abstract
The CLEF-2022 CheckThat! Lab Task 1 addresses the growing "infodemic" on social media by providing a robust framework for identifying check-worthy, verifiable, and harmful claims in tweets across six languages. This paper summarizes the participation of 19 teams, highlighting the shift toward large-scale multilingual Transformer models like mT5 and GPT-3. While these models show promise, results indicate a significant performance gap when evaluated against professional fact-checking standards compared to crowdsourced data.
Introduction: The Social Media Triage Problem
During major global events such as the COVID-19 pandemic or national elections, journalists and policymakers are overwhelmed by the volume of social media data. Manual monitoring is no longer feasible. The 2022 CheckThat! Lab Task 1 aims to build an automated "triage" system to prioritize content for human review.
As shown in the pipeline above, the system categorizes tweets into four critical areas:
- Check-worthy: Claims that warrant investigation by professional fact-checkers.
- Verifiable: Distinguishing factual claims from mere opinions.
- Harmful: Identifying content potentially damaging to society.
- Attention-worthy: Highlighting specific claims that require immediate policy-level attention.
Methods and Architectures
The participating teams almost exclusively adopted Transformer-based architectures, moving away from traditional feature engineering. The technical landscape was dominated by multilingual approaches and advanced data strategies.
Key technical trends included:
- Multilingual Transformers: Models like mT5 (Multilingual T5) allowed for "knowledge transfer," where data from high-resource languages like English improved performance on lower-resource languages like Bulgarian.
- Zero-Shot Learning: Teams utilized GPT-3, fine-tuning on English and applying it directly to other languages.
- Data Augmentation: Techniques like back-translation were used to expand small datasets, creating synthetic training examples to improve model robustness.
- Advanced Regularization: Innovative methods like Manifold Mixup were explored to create smoother decision boundaries and better generalization.
Datasets and the "Professional Gap"
The authors released the CT-CWT-22 corpus, covering Arabic, Bulgarian, Dutch, English, Spanish, and Turkish, primarily focused on COVID-19 and politics. A critical finding emerged from the Spanish dataset, which was unique because it was annotated by professional journalists from the fact-checking organization Newtral.
Models performed significantly worse on the Spanish dataset than on others. This suggests a "Professional Gap": what a layperson (crowdsourced worker) deems check-worthy differs substantially from the editorial priorities of professional journalists. Aligning AI objectives with these professional standards remains a primary challenge.
Key Results and Insights
Performance varied by task and language:
- Check-worthiness (1A): Top F1-scores ranged from 0.571 (Spanish) to 0.801 (Turkish). The lower Spanish score reinforces the difficulty of matching professional editorial standards.
- Verifiability (1B): This was a more straightforward task, with accuracies reaching 0.839 in Bulgarian.
- Harmfulness (1C): Arabic proved the most challenging (F1 of 0.557), while English and Turkish saw higher success rates.
- Attention-worthy (1D): Bulgarian models achieved a high Weighted F1 of 0.915, whereas Arabic models struggled to beat random baselines.
Conclusion and Future Work
While Transformer models have advanced the state of the art, the task is far from solved. Future research must move beyond tweet text to incorporate external evidence (retrieval-augmented approaches) and user context (metadata and influence). Most importantly, closing the gap between automated systems and professional editorial criteria is essential for real-world deployment in newsrooms and policy offices.
Summary updated