NLPColab at FigNews 2024 Shared Task: Challenges in Bias and Propaganda Annotation for News Media

Sadaf Abdul Rauf, Huda Sarfraz, Saadia Nauman, Arooj Fatima, SadafZiafat SadafZiafat, Momina Ishfaq, Alishba Suboor, Hammad Afzal, Seemab LatifACL 2024

TL;DR

This paper details the NLPColab team's participation in the FIGNEWS 2024 shared task, focusing on annotating news for bias and propaganda related to the Gaza-Israel war. The team achieved top rankings, highlighting the need for more comprehensive annotation guidelines and the challenges in detecting subtle forms of bias and propaganda.

What Problem Does This Paper Address?

The proliferation of news and social media has made it increasingly difficult to discern objective reporting from biased or propagandistic content. This is particularly true in the context of complex geopolitical events like the Gaza-Israel conflict, where narratives can be heavily manipulated. The FIGNEWS 2024 shared task aimed to address this by creating a shared corpus and developing improved annotation guidelines for bias and propaganda detection across multiple languages. The challenge lies in creating guidelines that are detailed enough to capture nuanced forms of bias and propaganda, while also being practical for annotators to apply consistently.

Methods

The NLPColab team adopted a multi-faceted approach. They began by thoroughly studying existing definitions of bias and propaganda, then developed their own detailed interpretations of the FIGNEWS labels. This involved creating a categorization of bias and propaganda types, which served as the foundation for their annotation guidelines. For the annotation process itself, the team divided the task into batches, with each member annotating posts for both bias and propaganda. They implemented regular feedback sessions and review meetings to ensure consistency and refine the guidelines. To measure agreement, they used Cohen's Kappa score.

Datasets

The primary dataset used was the FIGNEWS 2024 corpus, which comprised news article headlines and advertising posts from Facebook. These posts covered the Gaza-Israel war (October 1, 2023 - January 31, 2024) and were available in English, Arabic, Hebrew, French, and Hindi. Machine translations into English and Arabic were provided. The corpus was shared in 15 batches of 1000 posts each. An IAA subset of 200 posts was also provided for inter-annotator agreement analysis.

Key Insights and Ideas

  • The Nuance of Bias and Propaganda: Detecting subtle forms of bias and propaganda is significantly more challenging than it appears, requiring linguistic proficiency, an understanding of socio-political contexts, and the ability to discern underlying agendas.
  • The Importance of Comprehensive Guidelines: The team's success and challenges underscored the critical need for detailed and comprehensive annotation guidelines to ensure consistent and accurate labeling.
  • Multidisciplinary Approach is Key: A team composed of individuals from diverse backgrounds (media studies, computer science, NLP) brought a broad range of perspectives, leading to richer discussions and a more nuanced understanding of bias and propaganda.
  • Translation's Impact: The paper raises an interesting question about whether machine translations of news articles amplify or de-emphasize bias and propaganda, a factor that could influence annotation.
  • Self-Reflection for Objectivity: Given the sensitive nature of the conflict, annotators engaged in conscious self-reflection to mitigate personal biases and maintain the integrity of their annotations.

Key Results

The NLPColab team achieved strong performance in the FIGNEWS 2024 shared task:

  • Propaganda Annotation: Ranked first in all four evaluation tracks (Guidelines, IAA Quality, Quantity, Consistency).
  • Bias Annotation: Ranked first in the Guidelines and IAA Quality tracks, and second in the Quantity and Consistency tracks.
  • Inter-Annotator Agreement (IAA): Achieved an overall average of 0.775 for the Bias subtask and 0.679 for the Propaganda subtask, indicating a fairly healthy level of agreement among annotators.

Limitations and Future Work

  • Fact-Checking Challenges: The paper acknowledges that fact-checking was a challenging part of the process, requiring improvement. The team relied on a curated list of news sources but noted the difficulty in consistently verifying information.
  • Automation Potential: The authors suggest that leveraging technology to automate parts of the annotation process, such as initial tagging or flagging, could increase efficiency and allow annotators to focus on more subjective and complex decisions.
  • Cultural Variations: Subtle bias and propaganda can vary significantly across different cultures, which complicates the annotation process and suggests a need for culturally sensitive guidelines.

Summary updated

BibTeX
@inproceedings{rauf2024nlpcolab,
  author = {Sadaf Abdul Rauf and Huda Sarfraz and Saadia Nauman and Arooj Fatima and SadafZiafat SadafZiafat and Momina Ishfaq and Alishba Suboor and Hammad Afzal and Seemab Latif},
  title = {{NLPColab at FigNews 2024 Shared Task: Challenges in Bias and Propaganda Annotation for News Media}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure