TL;DR
This paper introduces a new, large-scale, multilingual dataset of 12,000 Facebook posts annotated for bias and propaganda, specifically focusing on the Israeli War on Gaza. This resource aims to advance the detection of misinformation in social media across Arabic, Hebrew, English, French, and Hindi, addressing a critical need for nuanced understanding of online discourse.
What Problem Does This Paper Address?
The proliferation of bias and propaganda on social media poses a significant challenge to understanding and navigating online information. Detecting these forms of misinformation is crucial, as they can polarize public opinion, fuel extremism, and erode trust in democratic processes. While previous efforts have focused on propaganda detection, there's a growing need for comprehensive, multilingual datasets that capture the nuances of bias and propaganda in the context of real-world events, especially those with significant geopolitical implications like the Israeli War on Gaza. This paper addresses this gap by creating a richly annotated dataset to support research in this area.
Methods
The authors developed a detailed annotation methodology based on established guidelines from the UN and EU for defining "bias." They introduced specific classifications for bias, including explicit, implicit, and vague types, to capture subtle forms of prejudice. For propaganda, they adopted existing classifications and added a crucial layer of categorization: "Propaganda must be deleted," "Propaganda may be deleted," and "Propaganda not to be deleted," to assess the potential harm of the content. The annotation process involved 10 graduate students specializing in Law, who underwent rigorous training and participated in multiple workshops to ensure consistency and accuracy. Inter-Annotator Agreement (IAA) was meticulously measured using F1-score and Cohen's Kappa to validate the quality of the annotations.
Datasets
The core contribution is the FigNews 2024 Shared Task dataset, comprising 12,000 Facebook posts. This dataset is multilingual, covering Arabic, Hebrew, English, French, and Hindi, with 2,400 posts per language. The posts were collected from verified Facebook accounts between October 7, 2023, and January 31, 2024, using "Gaza" as a query keyword. This dataset is novel in its multilingual scope and its specific focus on the recent conflict, building upon prior work like SemEval-2020 Task 11 and TSHP-17 but with a broader language coverage and a dual focus on both bias and propaganda.
Key Insights and Ideas
- Nuanced Bias Classification: The introduction of "explicit," "implicit," and "vague" bias types is a significant step beyond simple binary classification, allowing for a more granular understanding of how bias manifests in social media.
- Harm-Aware Propaganda Labeling: Categorizing propaganda based on its potential for harm ("must be deleted," "may be deleted," "not to be deleted") is a practical and important addition, moving beyond simple detection to assessing the real-world impact of misinformation.
- Multilingual Focus is Crucial: The study highlights that bias and propaganda can manifest differently across languages, with French posts showing a higher proportion of bias against Palestine and propaganda requiring deletion. This underscores the necessity of multilingual datasets for a comprehensive understanding of online discourse.
- Law Students as Annotators: Utilizing law students provided a unique perspective, potentially bringing a deeper understanding of legal and ethical implications of bias and propaganda, which could contribute to more robust annotations.
Key Results
The Sina team achieved strong results in the FIGNEWS 2024 shared task, securing third and second place in the IAA Quality and Quantity tracks for Bias and Propaganda, respectively. The overall Inter-Annotator Agreement (IAA) was 80.8% for bias and 70.15% for propaganda, indicating a "very good" and "good" agreement, respectively. The dataset analysis revealed that approximately 27% of posts were biased against Palestine, with a significant portion originating from French posts. For propaganda, 31% of posts were classified as such, with French posts again showing a higher prevalence, particularly for content deemed "Propaganda must be deleted."
Limitations and Future Work
The authors acknowledge the inherent subjectivity in identifying bias and propaganda, as well as the sensitivity of the dataset's content. Machine translation can also introduce bias, which they attempted to mitigate through careful review of Arabic and English translations. Future work planned by the authors includes expanding the corpus to cover more critical events of the war and leveraging neural and large language models to automate the detection of bias and propaganda.
Summary updated