Abstract
With the continuing spread of misinformation and disinformation online, it is of increasing importance to develop combating mechanisms at scale in the form of automated systems that support multiple languages. One task of interest is claim veracity prediction, which can be addressed using stance detection with respect to relevant documents retrieved online. To this end, we present our new Arabic Stance Detection dataset (AraStance) of 4,063 claim–article pairs from a diverse set of sources comprising three fact-checking websites and one news website. AraStance covers false and true claims from multiple domains (e.g., politics, sports, health) and several Arab countries, and it is well-balanced between related and unrelated documents with respect to the claims. We benchmark AraStance, along with two other stance detection datasets, using a number of BERT-based models. Our best model achieves an accuracy of 85% and a macro F1 score of 78%, which leaves room for improvement and reflects the challenging nature of AraStance and the task of stance detection in general.
Introduction
Automated fact-checking has become a critical tool in the fight against digital misinformation. A central component of this pipeline is stance detection: the ability of a system to determine the relationship between a claim and a supporting or refuting document. Specifically, a model must decide if an article agrees with, disagrees with, discusses, or is unrelated to a given claim.
While English-language stance detection has seen significant progress, Arabic presents a more complex landscape due to its rich dialectal diversity and a historical lack of high-quality, multi-domain datasets. Previous efforts often focused on narrow topics or relied on synthetic data. AraStance addresses these gaps by providing a human-annotated, multi-country dataset that reflects real-world fact-checking scenarios.
The AraStance Dataset
AraStance consists of 4,063 claim-article pairs derived from 910 unique claims. The dataset was constructed using a rigorous three-step process:
- Claim Collection: False claims were sourced from three prominent fact-checking platforms (ARANEWS, DABEGAD, and NORUMERS) based in the UAE, Egypt, and Saudi Arabia. True claims were collected from Reuters Arabic to ensure a balanced veracity profile.
- Document Retrieval: To simulate real-world conditions, the researchers used a "multi-query" approach with the Google Search API. By generating variations of each claim—extracting named entities and core verb phrases—they were able to retrieve a balanced mix of relevant and unrelated articles.
- Stance Annotation: Three native Arabic-speaking graduate students manually labeled the pairs. The annotation process achieved a Fleiss kappa of 0.67, indicating substantial agreement, though the "discuss" label proved to be the most nuanced and difficult to distinguish from "agree."
Benchmarking and Results
The researchers benchmarked AraStance against two existing datasets—Baly et al. (2018) and Khouja (2020)—using four transformer-based models: mBERT, ArabicBERT, ARBERT (Modern Standard Arabic), and MARBERT (Arabic tweets).
Key Findings
- ARBERT's Superiority: ARBERT consistently outperformed other models, achieving 85% accuracy and a 78% macro F1 score on AraStance. This success is attributed to its training on Modern Standard Arabic (MSA) and its "next sentence prediction" objective, which is highly effective for modeling the relationship between two distinct text segments.
- The "Discuss" Challenge: While models were highly effective at identifying "unrelated" documents (F1 > 0.90), they struggled with the "discuss" class (F1 scores between 0.53 and 0.60). This suggests that neutral reporting and contextual discussion require more sophisticated reasoning than simple agreement or contradiction.
- Generalization: Models trained on AraStance demonstrated better cross-domain generalization than those trained on more specialized, single-source datasets.
Conclusion and Future Directions
AraStance provides a robust foundation for Arabic stance detection, yet the results highlight ongoing challenges. The dominance of the "unrelated" class and the difficulty of nuanced labels like "discuss" remain hurdles for current BERT-based architectures.
Future work will focus on addressing class imbalance through weighted loss functions and exploring methods to handle long-form articles, as current models are limited by a 512-token truncation. By improving these automated systems, the research community can better support human fact-checkers in identifying and debunking misinformation across the Arab world.
Summary updated