JusticeLeague at FIGNEWS 2024 Shared Task: Innovations in Bias Annotation

Amr Saleh, Huda Mohamed, Hager SayedACL 2024

TL;DR

This paper details a novel approach to annotating news articles for bias, specifically focusing on the Gaza-Israel conflict. The team developed a granular annotation scheme and rigorous guidelines, emphasizing the use of concrete examples to ensure consistency and achieve high inter-annotator agreement, contributing a valuable dataset to the field of media bias detection.

What Problem Does This Paper Address?

The media's portrayal of conflicts, particularly the Gaza-Israel conflict, is often fraught with bias, influencing public opinion and policy. Accurately identifying and categorizing this bias is crucial for promoting media literacy and unbiased journalism. While previous work has explored hate speech and stance detection, there's a need for a more systematic and nuanced approach to categorizing bias in news articles, especially concerning specific entities involved in complex geopolitical events. This paper addresses the challenge of creating a reliable and consistent dataset for bias annotation in news articles related to the Gaza-Israel conflict.

Methods

The core of the methodology lies in the development of comprehensive and granular annotation guidelines. The team defined several bias categories, including "Unbiased," "Biased against Palestine," "Biased against Israel," "Biased against both Palestine and Israel," "Biased against others," "Unclear," and "Not Applicable." A key innovation was the extensive use of label-specific examples to illustrate each category, which proved instrumental in clarifying subtle biases and ensuring consistent application of the guidelines. The annotation process itself was collaborative, utilizing tools like Google Drive and Google Sheets for efficient data management and communication. To assess reliability, an Inter-Annotator Agreement (IAA) analysis was conducted, focusing on qualitative review and discussion of discrepancies rather than solely relying on quantitative metrics.

Datasets

The paper does not explicitly name a pre-existing dataset used for training. Instead, it describes the creation of a new dataset through the annotation of news articles for the FIGNEWS 2024 Shared Task. The team annotated approximately 2,400 data points across two main batches and six IAA batches.

Key Insights and Ideas

  • Granularity is Key: Simply labeling content as "biased" is insufficient. Categorizing bias towards specific entities (e.g., "Biased against Palestine") provides a deeper understanding of the nature and extent of media bias.
  • Examples are Powerful: The extensive use of concrete examples within the annotation guidelines was crucial for achieving high inter-annotator agreement and ensuring consistent labeling, especially for subtle biases.
  • Qualitative IAA Matters: While quantitative IAA metrics are standard, a qualitative approach involving manual review and discussion of annotations can effectively validate the reliability of the annotation process and the clarity of the guidelines.
  • Collaboration Enhances Efficiency: Utilizing collaborative tools like Google Drive and Sheets significantly streamlined the annotation process, facilitating efficient data management and real-time communication among annotators.

Key Results

The team achieved impressive Inter-Annotator Agreement (IAA) metrics, including a Kappa of 64.4 and an Accuracy of 83.7%. Their methodology led to a second-place achievement in the IAA-Quality track of the FIGNEWS 2024 Shared Task. Qualitatively, the analysis revealed that while a significant portion of content was unbiased (42.8%), there was a notable prevalence of content biased against Palestine (29.2%), particularly in Hebrew-language sources. The varying distributions of bias across different languages underscored the need for multilingual approaches in bias detection.

Limitations and Future Work

The authors acknowledge that their IAA metrics, while strong, were lower when comparing across different teams (internal IAA vs. external). This highlights the inherent subjectivity in bias perception and the ongoing need for refining annotation guidelines and methodologies. Future work could involve developing more sophisticated machine learning models for automated bias detection, building upon the insights gained from their manual annotation process. Further investigation into the relationship between bias and propaganda is also suggested, as is the potential for creating more robust and nuanced datasets for this critical area of research.

Summary updated

BibTeX
@inproceedings{saleh2024justiceleague,
  author = {Amr Saleh and Huda Mohamed and Hager Sayed},
  title = {{JusticeLeague at FIGNEWS 2024 Shared Task: Innovations in Bias Annotation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure