The CyberEquity Lab at FIGNEWS 2024 Shared Task: Annotating a Corpus of Facebook Posts to Label Bias and Propaganda in Gaza-Israel War Coverage in Five Languages

Mohammed Helal, Radi Jarrar, Mohammed Alkhanafseh, Abdallah Karakra, Ruba AwadallahACL 2024

TL;DR

This paper details the CyberEquity Lab's approach to annotating a large corpus of Facebook posts for bias and propaganda related to the Gaza-Israel war. They developed detailed annotation guidelines and achieved moderate to fair inter-annotator agreement, highlighting the challenges and nuances of identifying these phenomena in short-form social media content across multiple languages.

What Problem Does This Paper Address?

The ongoing Gaza-Israel conflict is heavily influenced by information warfare, with traditional media and social media platforms often acting as conduits for bias and propaganda. This makes it difficult for the public to form informed opinions. While detecting bias and propaganda in news articles has been explored, the challenge is amplified in the context of short, diverse social media posts, especially across multiple languages. This paper addresses the need for a robust, annotated dataset and clear guidelines to tackle this problem, specifically within the FIGNEWS 2024 Shared Task.

Methods

The core of the methodology lies in developing and applying a set of annotation guidelines for identifying bias and propaganda. The team iteratively refined these guidelines through extensive discussions, aiming for precision and practicality for short social media texts.

For bias, they adapted existing frameworks and developed specific criteria, including:

  • Dehumanization of a group: Attacking individuals based solely on their group identity.
  • Hate Speech: Abusive or threatening language expressing prejudice.
  • Double Standards: Applying different criteria to different groups.
  • Misinformation: Spreading false information to target a group.
  • Labeling and Name Calling: Using offensive labels to alienate groups.

For propaganda, they drew from established categories, selecting 12 that were deemed most relevant. Key propaganda techniques identified include:

  • Appeal to Authority: Citing a valid authority to support a claim.
  • Appeal to Fear/Prejudices: Instilling anxiety or panic.
  • Exaggeration/Minimization: Distorting the scale or severity of events.
  • Flag-waving: Appealing to nationalistic or group sentiment.
  • Loaded Language: Using emotionally charged words.
  • Slogans: Short, impactful phrases often used for emotional appeal.

A crucial aspect of their approach is the decision to consider any biased post as propaganda, acknowledging the overlap. However, they also note that an unbiased post can still contain propaganda. The annotation process involved team members logging into shared files, reading posts in their preferred language, and systematically applying the guidelines. They also emphasized the importance of inter-annotator agreement (IAA) to ensure consistency.

Datasets

The paper focuses on the FIGNEWS 2024 Shared Task corpus, which consists of approximately 15,000 Facebook posts. These posts were collected from October 1st, 2023, to January 31st, 2024, by searching for the keyword "Gaza" in five languages: Arabic, Hebrew, French, Hindi, and English. The posts were translated into Arabic and English using translation tools for annotation. The CyberEquity Lab team manually annotated around 3,000 of these posts, with 10% set aside for IAA analysis.

Key Insights and Ideas

  • The Overlap is Significant: The team found that bias and propaganda often overlap, with biased posts frequently being considered propagandistic. This highlights the interconnected nature of these phenomena in online discourse.
  • Short-form Content is Tricky: Annotating bias and propaganda in short social media posts presents unique challenges compared to longer news articles, requiring concise and precise guidelines.
  • Context is King (and Difficult): Understanding the nuances of political discourse and cultural context is crucial for accurate annotation, especially given the team's Palestinian background and immersion in the conflict.
  • Language Diversity Adds Complexity: Annotating across five languages, even with translations, introduces potential for misinterpretation and requires careful consideration of linguistic subtleties.
  • The "Unbiased" Label is Nuanced: The paper clarifies that a post can be labeled "unbiased" even if it reports on biased information, as long as the author does not inject their own subjective opinion. This distinction is vital for accurate classification.

Key Results

The CyberEquity Lab team successfully annotated approximately 3,000 posts from the FIGNEWS 2024 corpus. Their performance in the propaganda subtask placed them third out of sixteen teams, based on quantity of data points, quality of results (IAA agreement), and consistency score (cross-team Macro F1 average).

Inter-annotator agreement scores, measured by Cohen's Kappa, indicated:

  • Bias Annotation: Moderate agreement (scores ranging from 0.39 to 0.56).
  • Propaganda Annotation: Slight to moderate agreement (scores ranging from 0.15 to 0.48).

The authors interpret these scores, noting that bias was generally easier to identify and correlate with news than propaganda, which has a larger number of categories and can be more challenging to classify consistently.

Limitations and Future Work

Limitations: * Subjectivity in Annotation: Despite detailed guidelines, some level of subjectivity is inherent in identifying bias and propaganda, as reflected in the moderate IAA scores. * Translation Accuracy: While translations were used, potential inaccuracies or loss of nuance could affect annotation. * Limited Corpus Size for ML: While 3,000 annotated posts are deemed sufficient for training, larger datasets would undoubtedly lead to more robust ML models.

Future Work: * The paper suggests that the annotated corpus can be used to train highly accurate NLP models and evaluate clustering models for bias and propaganda detection. * Further refinement of annotation guidelines and exploration of more sophisticated annotation methodologies could improve IAA. * Expanding the corpus to include more languages or different social media platforms would broaden the scope of analysis.

Summary updated

BibTeX
@inproceedings{helal2024cyberequity,
  author = {Mohammed Helal and Radi Jarrar and Mohammed Alkhanafseh and Abdallah Karakra and Ruba Awadallah},
  title = {{The CyberEquity Lab at FIGNEWS 2024 Shared Task: Annotating a Corpus of Facebook Posts to Label Bias and Propaganda in Gaza-Israel War Coverage in Five Languages}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure