BiasGanda at FIGNEWS 2024 Shared Task: A Quest to Uncover Biased Views in News Coverage

Al Manar Al Wardi, Blqees Al Busaidi, Malath Al-Sibani, Hiba Salim Muhammad Al-Siyabi, Najma Al ZidjalyACL 2024

TL;DR

This paper tackles the challenge of identifying biased language in news coverage of the Gaza-Israel war by developing and applying a detailed annotation scheme. The authors found that the majority of the analyzed news content exhibited bias against Palestine, highlighting the pervasive nature of media bias across different languages.

What Problem Does This Paper Address?

The paper addresses the critical issue of media bias, specifically within news coverage of the complex and often volatile Gaza-Israel conflict. Identifying and understanding how news outlets frame events is crucial for informed public discourse. Existing research has explored methods for detecting bias, but a comprehensive, shared dataset and annotation framework for recent events like the Gaza-Israel war were lacking. This work aims to fill that gap by creating a meticulously annotated corpus and analyzing the linguistic tendencies used by media outlets when reporting on this specific conflict.

Methods

The core of this research lies in the development of a robust annotation methodology. The team established a detailed codebook with definitions for seven bias categories: Unbiased, Biased against Palestine, Biased against Israel, Biased against Others, Biased against both Palestine and Israel, Unclear, and Not Applicable. This codebook was informed by existing literature on bias detection and refined through weekly meetings and discussions with a supervisor. The annotation process involved four female Omani annotators who independently coded examples from the FIGNEWS24 shared task dataset. To ensure consistency, they utilized Inter-Annotator Agreement (IAA) methodologies, where annotators independently coded specific texts without prior discussion. The team also focused on distinguishing between framing bias (presenting a subject from a specific perspective) and epistemological bias (subtly implying propositions as true or false).

Datasets

The primary dataset used in this study was provided by the FIGNEWS24 shared task, specifically focusing on news coverage of the Gaza-Israel war of 2023-2024. This dataset was divided into "MAIN" and "IAA" sheets for annotation. The paper does not explicitly mention the creation of a new benchmark dataset, but rather the annotation of an existing one for the shared task.

Key Insights and Ideas

  • Pervasive Bias Against Palestine: The most striking finding is that the majority of the analyzed news examples were classified as "Biased against Palestine." This suggests a significant imbalance in how the conflict is being reported.
  • One-Sided Language: The study observed that one-sided terms were predominantly used to describe the October 7th event, contributing to the identified bias.
  • Language-Specific Nuances: While bias against Palestine was consistent across languages, the distribution of other bias categories varied. For instance, Hebrew sources showed the lowest percentage of "Biased against Israel," likely due to the dominance of Israeli media outlets in that language category.
  • Unbiased Content Interpretation: The researchers found that a significant portion of content was categorized as unbiased, partly due to their methodology of considering reported speech and quotes based on how the media outlet presented them, rather than the content of the quote itself.
  • "Not Applicable" as an Indicator: The "Not Applicable" category, used for irrelevant texts, was least used, suggesting a focused data collection for the task. However, its low occurrence across languages might also point to potential data collection process issues.

Key Results

The team annotated 2,200 data points in total. Their primary finding was that "Biased against Palestine" was the most frequent category across all languages. The "Not Applicable" category was the least used. In terms of performance metrics, the team achieved a Kappa score of 31.0 in the IAA Quality Track, indicating good consistency and adherence to guidelines within their team. They also achieved a Macro F1 Average score of 29.5 in the Centrality Track, ranking 4th overall, which reflects their consistency and adherence to guidelines across all participating teams. Their final rank was 6th, indicating strong performance in the FIGNEWS shared task.

Limitations and Future Work

The paper acknowledges that the "Not Applicable" category's low percentage across languages might suggest a potential error in the data collection process. The authors also note that while their methodology considered reported speech and quotes, this approach might have influenced the categorization of unbiased content. Future work could involve exploring more nuanced methods for detecting bias, potentially incorporating deeper semantic analysis or cross-lingual bias detection techniques. Further investigation into the reasons behind the language-specific variations in bias distribution would also be valuable.

Summary updated

BibTeX
@inproceedings{wardi2024biasganda,
  author = {Al Manar Al Wardi and Blqees Al Busaidi and Malath Al-Sibani and Hiba Salim Muhammad Al-Siyabi and Najma Al Zidjaly},
  title = {{BiasGanda at FIGNEWS 2024 Shared Task: A Quest to Uncover Biased Views in News Coverage}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure