Uot1 at FIGNEWS 2024 Shared Task: Labeling News Bias

Abdusalam Nwesri, Mai Elbaabaa, Fatima Lashihar, Fatma AlalosACL 2024

TL;DR

This paper details the University of Tripoli's participation in the FIGNEWS 2024 shared task, focusing on developing annotation guidelines and a tool to detect bias in news articles about the Israeli-Palestinian conflict. Their efforts resulted in a ranked performance and highlighted the complexities of multilingual bias annotation.

What Problem Does This Paper Address?

The paper tackles the critical issue of bias in news reporting, particularly concerning sensitive geopolitical conflicts like the Israeli-Palestinian conflict. While NLP has made strides in bias detection for English news, there's a significant gap in applying these techniques to other languages and in multi-label contexts. This lack of resources hinders the development of AI models that can accurately identify and understand nuanced media bias across diverse linguistic and cultural landscapes. The authors aim to address this by creating a robust, multilingual dataset and annotation framework.

Methods

The core of their methodology involved developing a set of annotation guidelines for identifying bias in news articles. They started with initial labels provided by the FIGNEWS 2024 organizers and iteratively refined them based on their annotation experience. This iterative process led to the addition of new labels like "Biased in Favour of Palestine" and "Biased in Favour of Israel" to better capture specific directional biases. To facilitate the annotation process, they developed a custom in-house tool that allowed annotators to view articles in their preferred language (Arabic being a focus) and efficiently label them. They also emphasized team coordination and training to ensure consistency and inter-annotator agreement.

Datasets

The primary dataset used was provided by the FIGNEWS 2024 shared task organizers, consisting of 15,000 news articles related to the 2023-2024 Gaza-Israel war. This dataset was divided into batches, with each batch containing an "Inter-Annotator Agreement" (IAA) subset and a larger "MAIN" subset. The articles were available in multiple languages (English, Arabic, Hebrew, French, Hindi) and machine-translated into English and Arabic. The authors annotated a total of 3,900 articles.

Key Insights and Ideas

  • Iterative Guideline Refinement is Crucial: The initial set of labels proved insufficient, necessitating the addition of more specific "in favor of" labels to accurately capture directional bias, especially in complex conflicts.
  • Multilingual Bias Annotation is Inherently Challenging: Translation nuances and the dual meanings of words (e.g., "Hamas") can lead to significant annotation discrepancies, underscoring the need for careful consideration of linguistic context.
  • Custom Tools Enhance Efficiency: Developing an in-house annotation tool significantly improved the annotation process, especially for handling large volumes of text and multiple languages.
  • Inter-Annotator Agreement is a Key Indicator of Guideline Quality: While the paper notes a moderate disagreement, the process of discussing and resolving these disagreements was vital for refining the guidelines and achieving a shared understanding of bias.

Key Results

The University of Tripoli team achieved a strong performance in the FIGNEWS 2024 shared task:

  • They ranked 4th in the quality track using the macro F1 average measure.
  • They ranked 6th in the centrality track using the macro F1 average measure.
  • They achieved the 4th best kappa coefficient in the centrality track.
  • Their bias annotation guidelines were ranked 9th.
  • They successfully annotated 3,900 articles using their custom tool and refined guidelines.

Limitations and Future Work

The authors acknowledge several limitations:

  • Tedious Annotation Process: The task is described as tedious and demanding, requiring sustained effort and concentration.
  • Translation Challenges: Reliance on machine translation introduced difficulties and potential misinterpretations.
  • Initial Label Limitations: The initial set of labels provided by the organizers required significant refinement.
  • Re-annotation Necessity: They were instructed to remove their added labels and re-annotate, highlighting the importance of adhering to standardized guidelines.

For future work, the authors plan to further refine and expand their guidelines and tools, aiming to enhance the capacity of researchers and practitioners to identify and mitigate bias in media narratives.

Summary updated

BibTeX
@inproceedings{nwesri2024uot,
  author = {Abdusalam Nwesri and Mai Elbaabaa and Fatima Lashihar and Fatma Alalos},
  title = {{Uot1 at FIGNEWS 2024 Shared Task: Labeling News Bias}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure