TL;DR
This paper introduces SCRum-9, the largest multilingual dataset for rumour stance classification across nine languages, designed to capture annotator uncertainty. They benchmark large language models (LLMs) and smaller models, finding that while LLMs excel in zero-shot and few-shot learning, fine-tuning smaller models with LLM-generated synthetic data offers a computationally efficient and competitive alternative.
What Problem Does This Paper Address?
The spread of misinformation and rumors on social media is a global problem, and understanding user stances (support, refute, question) is crucial for combating it. However, existing datasets for rumour stance classification are predominantly English-centric, failing to capture the multilingual nature of online discourse. While some multilingual stance classification datasets exist, they often focus on political opinions or social issues and use different task framings, making them unsuitable for rumour analysis. This gap limits the development and evaluation of models that can effectively handle rumors across diverse linguistic and cultural contexts.
Methods
The authors introduce SCRum-9, a new benchmark dataset for multilingual rumour stance classification. Key innovations include:
- Multilingual Coverage: Nine languages are included, significantly expanding beyond existing datasets.
- Rich Annotations: Beyond the standard stance labels (Confirming, Rejecting, Questioning, Commenting), annotators provided a confidence score and a second-choice label to capture uncertainty.
- Fact-Checked Claims: The dataset links replies to specific fact-checked claims, enabling deeper analysis.
- LLM and MLM Benchmarking: They evaluate five instruction-tuned LLMs and two multilingual masked language models (MLMs) using In-Context Learning (ICL) and fine-tuning approaches.
- Synthetic Data Generation: A novel aspect is the exploration of using LLMs to generate synthetic multilingual data for fine-tuning MLMs, aiming for a more efficient and scalable solution.
- Translation and Alignment Strategies: For improving ICL in non-English languages, they investigate machine translating inputs and using language alignment signals in prompts.
Datasets
- SCRum-9: The primary contribution, a new benchmark dataset containing 7,516 reply tweets across nine languages (Czech, German, English, Spanish, French, Hindi, Polish, Portuguese, Russian). It links to 2,156 distinct rumours from fact-checking websites and includes confidence and second-choice annotations.
- RumourEval 2017/2019: Used for few-shot ICL demonstrations and fine-tuning MLMs on English data.
Key Insights and Ideas
- Annotator Uncertainty is Key: The inclusion of second-choice labels and confidence scores reveals that rumour stance classification is inherently subjective. Models that can account for this uncertainty perform better, especially in ambiguous cases.
- Translation is a Strong Baseline for ICL: For non-English languages, machine translating inputs into English significantly boosts LLM performance in zero-shot ICL, especially for languages with good translation quality.
- Few-Shot ICL Outperforms Zero-Shot: Even with limited examples, few-shot ICL consistently improves performance over zero-shot, highlighting LLMs' ability to transfer knowledge across languages.
- Synthetic Data is a Game Changer for MLMs: Fine-tuning MLMs on LLM-generated synthetic multilingual data can achieve performance comparable to, or even exceeding, the zero-shot performance of the LLMs themselves, while being much more computationally efficient. This is particularly impactful for LLMs with weaker baseline ICL capabilities.
- LLM ICL Performance Predicts Synthetic Data Quality: LLMs that perform better in ICL tend to generate more effective synthetic data for fine-tuning MLMs.
Key Results
- LLM Performance Varies: LLMs show significant performance disparities across languages, generally performing best on English. Gemma demonstrated the most consistent cross-lingual performance among the LLMs tested.
- Translation vs. Alignment: Machine translating target input into English generally improved ICL performance more than language alignment signals, though Gemma was an exception.
- Fine-tuning MLMs: Fine-tuning XLM-R on synthetic data generated by LLMs like Qwen and Gemma achieved competitive results, often surpassing baseline zero-shot ICL performance of the LLMs themselves. For example, fine-tuning XLM-R on Qwen's synthetic data for German yielded a wF2 score of 0.43, significantly better than Qwen's baseline ICL of 0.21.
- Model vs. Human Uncertainty: When models predicted a second-choice label that aligned with human annotators' second choices, it indicated the models were capturing plausible interpretations rather than just making errors. This led to substantial improvements in evaluation metrics when considering second-choice labels.
Limitations and Future Work
- Annotator Availability: While SCRum-9 uses multiple annotators, budget constraints limited it to two or three per example. Future work could involve more annotators for even more robust consensus.
- Label Aggregation Methods: The paper explored various aggregation methods but did not fully leverage LLMs' potential to predict label distributions. Future research could investigate LLMs' ability to predict such distributions.
- Synthetic Data Quality Evaluation: The authors acknowledge the lack of native speakers prevented a thorough human evaluation of the generated synthetic data's lexical diversity and stance accuracy. Future work could explore this correlation.
- Claim Verification: SCRum-9's link to fact-checked claims opens avenues for claim verification research, which was not explored in this paper.
- Low-Resource Languages: The current dataset focuses on relatively high-resource languages. Future work should aim to create datasets for low-resource languages to promote inclusivity.
Summary updated