LexiconLadies at FIGNEWS 2024 Shared Task: Identifying Keywords for Bias Annotation Guidelines of Facebook News Headlines on the Israel-Palestine 2023 War

Yousra El-Ghawi, Abeer Marzouk, Aya KhamisACL 2024

TL;DR

This paper tackles the challenge of automatically detecting bias in news headlines, particularly concerning the Israel-Palestine conflict. The authors developed annotation guidelines and a theoretical system that leverages keyword analysis and sentiment scoring to identify biased language, aiming to improve upon existing methods that often struggle with context and implicit bias.

What Problem Does This Paper Address?

Identifying bias in news, especially on social media, is a significant challenge for both humans and machines. Existing datasets for bias detection are often limited in scope, lack linguistic nuance, or focus on document-level bias rather than the subtle language used within headlines. This paper addresses the need for better methods to automatically detect bias, particularly in highly sensitive and politically charged contexts like the Israel-Palestine conflict, where neutrality is difficult and implicit bias is common. The authors highlight that social media's algorithmic amplification and user interaction further complicate bias detection, making it crucial to understand how language is used to influence specific audiences.

Methods

The core of this paper lies in the development of annotation guidelines for identifying bias in news headlines. The authors employed a discourse-pragmatic approach, focusing on both sentence-level and token-level analysis. They created a lexicon of keywords related to sentiment, events, and targeted language specific to the Israel-Palestine conflict. This lexicon was used to guide manual annotation and to develop a theoretical system for bias classification. Their methodology involved:

  • Pilot Annotations: Initial annotation of a subset of headlines to identify recurring themes and words.
  • Keyword Lexicon Development: Compiling keywords associated with sentiment markers, events, and targeted language.
  • Sentence and Token-Level Analysis: Evaluating bias at both the overall sentence level and by identifying specific bias-inducing words.
  • Theoretical System Design: Proposing a system that extracts keywords, analyzes sentiment, and assigns bias labels based on entity (Palestine, Israel, Others).

Datasets

The primary dataset used in this research is from the FIGNEWS Subtask 1: Bias Annotation. This dataset consists of approximately 13,500 multilingual headlines and their associated advertisement posts from Facebook, covering major events of the Israel-Palestine conflict in Arabic, English, Hebrew, French, and Hindi. The authors manually annotated two batches, totaling 1800 posts, for their study. They also conducted pilot annotations on an additional 200 headlines per annotator to ensure consistency.

Key Insights and Ideas

  • Context is King (and Hard to Automate): The paper strongly emphasizes that identifying bias often requires significant contextual knowledge of surrounding events, which is difficult for automated systems to infer solely from keywords.
  • Implicit Bias is a Major Hurdle: Beyond overt "loaded" words, the research highlights how subtle linguistic choices, like the use of "captors" or "settlers," can implicitly convey bias.
  • Social Media Amplifies Bias: Hashtags and user engagement on platforms like Facebook can either mitigate or amplify bias, making it a complex factor in detection.
  • Keywords as Proxies for Bias: While not a complete solution, a carefully curated lexicon of keywords related to specific entities and sentiments can be a powerful tool for initial bias detection.
  • "Gatekeeping" and "Statement Bias" Dominate: The authors identified these two types of bias (selecting specific stories and injecting journalist opinion) as particularly prevalent in reporting on the conflict.

Key Results

The manual annotation of 1800 headlines revealed the following distribution of bias labels:

  • "Biased against Palestine": 34%
  • "Unbiased": 27%
  • "Biased against Israel": 16%
  • "Unclear": 8%
  • "Biased against both Palestine and Israel": 6%
  • "Biased against Others": 6%
  • "Not Applicable": 3%

The Inter-Annotator Agreement (IAA) scores indicated "fair" consistency for the overall bias label (Kappa score of 37.2) and a strong F1 score of 73.4 for bias detection. The developed guidelines achieved the highest F1 centrality score (33.1) among participating teams for bias annotation consistency.

Limitations and Future Work

The authors acknowledge several limitations:

  • Contextual Dependency: The reliance on external context for accurate bias labeling means that systems solely based on linguistic features may struggle.
  • Implicit Bias Complexity: Detecting subtle, implicit bias remains a significant challenge, even for human annotators.
  • Social Media Nuances: The impact of user engagement and irrelevant hashtags on bias detection requires further investigation.

Future work suggested includes:

  • Developing advanced machine learning methods that can learn context through user engagement data to detect minor instances of bias or clickbait.
  • Exploring how world context, user engagement, and media literacy influence overall bias detection.
  • Building more robust bias classification models that can handle diverse corpora and complex conflicts.

Summary updated

BibTeX
@inproceedings{elghawi2024lexiconladies,
  author = {Yousra El-Ghawi and Abeer Marzouk and Aya Khamis},
  title = {{LexiconLadies at FIGNEWS 2024 Shared Task: Identifying Keywords for Bias Annotation Guidelines of Facebook News Headlines on the Israel-Palestine 2023 War}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure