SCRum-9: Multilingual Stance Classification over Rumours on Social Media

Yue Li, Jake Vasilakes, Zhixue Zhao, Carolina Scarton

TL;DR

This paper introduces SCRum-9, the largest multilingual dataset for rumour stance classification across nine languages, designed to capture annotator uncertainty. They benchmark large language models (LLMs) and smaller models, finding that while LLMs excel in zero-shot and few-shot learning, fine-tuning smaller models with LLM-generated synthetic data offers a computationally efficient and competitive alternative.

What Problem Does This Paper Address?

The spread of misinformation and rumors on social media is a global problem, and understanding user stances (support, refute, question) is crucial for combating it. However, existing datasets for rumour stance classification are predominantly English-centric, failing to capture the multilingual nature of online discourse. While some multilingual stance classification datasets exist, they often focus on political opinions or social issues and use different task framings, making them unsuitable for rumour analysis. This gap limits the development and evaluation of models that can effectively handle rumors across diverse linguistic and cultural contexts.

Methods

The authors introduce SCRum-9, a new benchmark dataset for multilingual rumour stance classification. Key innovations include:

  • Multilingual Coverage: Nine languages are included, significantly expanding beyond existing datasets.
  • Rich Annotations: Beyond the standard stance labels (Confirming, Rejecting, Questioning, Commenting), annotators provided a confidence score and a second-choice label to capture uncertainty.
  • Fact-Checked Claims: The dataset links replies to specific fact-checked claims, enabling deeper analysis.
  • LLM and MLM Benchmarking: They evaluate five instruction-tuned LLMs and two multilingual masked language models (MLMs) using In-Context Learning (ICL) and fine-tuning approaches.
  • Synthetic Data Generation: A novel aspect is the exploration of using LLMs to generate synthetic multilingual data for fine-tuning MLMs, aiming for a more efficient and scalable solution.
  • Translation and Alignment Strategies: For improving ICL in non-English languages, they investigate machine translating inputs and using language alignment signals in prompts.

Datasets

  • SCRum-9: The primary contribution, a new benchmark dataset containing 7,516 reply tweets across nine languages (Czech, German, English, Spanish, French, Hindi, Polish, Portuguese, Russian). It links to 2,156 distinct rumours from fact-checking websites and includes confidence and second-choice annotations.
  • RumourEval 2017/2019: Used for few-shot ICL demonstrations and fine-tuning MLMs on English data.

Key Insights and Ideas

  • Annotator Uncertainty is Key: The inclusion of second-choice labels and confidence scores reveals that rumour stance classification is inherently subjective. Models that can account for this uncertainty perform better, especially in ambiguous cases.
  • Translation is a Strong Baseline for ICL: For non-English languages, machine translating inputs into English significantly boosts LLM performance in zero-shot ICL, especially for languages with good translation quality.
  • Few-Shot ICL Outperforms Zero-Shot: Even with limited examples, few-shot ICL consistently improves performance over zero-shot, highlighting LLMs' ability to transfer knowledge across languages.
  • Synthetic Data is a Game Changer for MLMs: Fine-tuning MLMs on LLM-generated synthetic multilingual data can achieve performance comparable to, or even exceeding, the zero-shot performance of the LLMs themselves, while being much more computationally efficient. This is particularly impactful for LLMs with weaker baseline ICL capabilities.
  • LLM ICL Performance Predicts Synthetic Data Quality: LLMs that perform better in ICL tend to generate more effective synthetic data for fine-tuning MLMs.

Key Results

  • LLM Performance Varies: LLMs show significant performance disparities across languages, generally performing best on English. Gemma demonstrated the most consistent cross-lingual performance among the LLMs tested.
  • Translation vs. Alignment: Machine translating target input into English generally improved ICL performance more than language alignment signals, though Gemma was an exception.
  • Fine-tuning MLMs: Fine-tuning XLM-R on synthetic data generated by LLMs like Qwen and Gemma achieved competitive results, often surpassing baseline zero-shot ICL performance of the LLMs themselves. For example, fine-tuning XLM-R on Qwen's synthetic data for German yielded a wF2 score of 0.43, significantly better than Qwen's baseline ICL of 0.21.
  • Model vs. Human Uncertainty: When models predicted a second-choice label that aligned with human annotators' second choices, it indicated the models were capturing plausible interpretations rather than just making errors. This led to substantial improvements in evaluation metrics when considering second-choice labels.

Limitations and Future Work

  • Annotator Availability: While SCRum-9 uses multiple annotators, budget constraints limited it to two or three per example. Future work could involve more annotators for even more robust consensus.
  • Label Aggregation Methods: The paper explored various aggregation methods but did not fully leverage LLMs' potential to predict label distributions. Future research could investigate LLMs' ability to predict such distributions.
  • Synthetic Data Quality Evaluation: The authors acknowledge the lack of native speakers prevented a thorough human evaluation of the generated synthetic data's lexical diversity and stance accuracy. Future work could explore this correlation.
  • Claim Verification: SCRum-9's link to fact-checked claims opens avenues for claim verification research, which was not explored in this paper.
  • Low-Resource Languages: The current dataset focuses on relatively high-resource languages. Future work should aim to create datasets for low-resource languages to promote inclusivity.

Summary updated

Abstract

We introduce SCRum-9, the largest multilingual Stance Classification dataset for Rumour analysis in 9 languages, containing 7,516 tweets from X. SCRum-9 goes beyond existing stance classification datasets by covering more languages, linking examples to more fact-checked claims (2.1k), and including confidence-related annotations from multiple annotators to account for intra- and inter-annotator variability. Annotations were made by at least two native speakers per language, totalling more than 405 hours of annotation and 8,150 dollars in compensation. Further, SCRum-9 is used to benchmark five large language models (LLMs) and two multilingual masked language models (MLMs) in In-Context Learning (ICL) and fine-tuning setups. This paper also innovates by exploring the use of multilingual synthetic data for rumour stance classification, showing that even LLMs with weak ICL performance can produce valuable synthetic data for fine-tuning small MLMs, enabling them to achieve higher performance than zero-shot ICL in LLMs. Finally, we examine the relationship between model predictions and human uncertainty on ambiguous cases finding that model predictions often match the second-choice labels assigned by annotators, rather than diverging entirely from human judgments. SCRum-9 is publicly released to the research community with potential to foster further research on multilingual analysis of misleading narratives on social media.
BibTeX
@inproceedings{li2025scrum,
  author = {Yue Li and Jake Vasilakes and Zhixue Zhao and Carolina Scarton},
  title = {{SCRum-9: Multilingual Stance Classification over Rumours on Social Media}},
  booktitle = {myPublications at the University of Sheffield},
  year = {2025},
  url = {https://openreview.net/forum?id=gLNaLPc5H7},
}
Figure