TL;DR
This paper introduces a novel annotation methodology for identifying bias in media narratives surrounding the Gaza War, specifically for the FIGNEWS 2024 Shared Task. The team developed a prescriptive annotation guideline, refined through iterative discussions, and explored an additional layer of "narrative frames" to better understand how bias is conveyed.
What Problem Does This Paper Address?
News media, while often aiming for objectivity, can subtly or overtly exhibit bias, influencing public perception, especially in politically charged contexts like the Gaza War. Identifying this bias is crucial but challenging due to its nuanced manifestations in word choice, framing, and selective reporting. The FIGNEWS 2024 Shared Task aims to address this by focusing on the intricate layers of bias and double standards present in news articles and social media posts related to the 2023 Gaza War. This paper details the annotation process for this task, contributing to a deeper understanding of how conflict narratives are framed and potentially manipulated.
Methods
The authors employed a prescriptive paradigm for developing their annotation guidelines, meaning the guidelines were iteratively refined through a structured discussion phase. This involved annotators identifying edge cases, discussing them, and converging on resolutions to ensure consistent labeling. They also introduced an optional narrative frames annotation layer. Frames are defined as conceptual building blocks of a narrative that support either a pro-Israeli or pro-Palestinian perspective. These frames were intended to aid in bias identification, even when explicit bias labels were not immediately apparent. The team annotated 2000 instances for bias and frames.
Datasets
The paper describes the dataset used for the FIGNEWS 2024 Shared Task, which consists of a curated multilingual collection of news article headlines and Facebook posts from news organizations. The texts cover the 2023 Gaza War and are in five languages: English, Arabic, Hebrew, French, and Hindi. The dataset was annotated by the authors' team.
Key Insights and Ideas
- The Subjectivity of Bias: The inter-annotator agreement (IAA) score of 0.420 (Krippendorff's alpha) highlights the inherent subjectivity and difficulty in consistently identifying bias, even with detailed guidelines.
- Frames as a Lens for Bias: The introduction of narrative frames proved to be a valuable tool, helping annotators identify bias even when it wasn't explicitly stated, by revealing the underlying conceptual frameworks being used. For example, the "IDF glorification" frame can indicate bias against Palestine without directly mentioning Palestine.
- Language-Specific Bias Patterns: The analysis revealed significant differences in bias distribution across languages, with Hebrew texts showing the highest percentage of bias against Palestine and French texts showing the most bias against Israel.
- The Nuance of "Unbiased": The paper clarifies that "unbiased" in their context means "lacking an explicit textual bias," acknowledging that bias can still exist through selective reporting or emphasis, which is not fully captured by their current annotation scheme.
Key Results
- The team achieved an inter-annotator agreement (IAA) of 0.420 (Krippendorff's alpha) for the bias annotation task.
- Their analysis showed that 52% of the annotated posts were unbiased, 42% were biased against Palestine, 5% were biased against Israel, and 3% were biased against both. 16% were unclear or not applicable.
- Hebrew texts exhibited the highest percentage of bias against Palestine, while French texts showed the most bias against Israel. English texts were found to be mostly unbiased.
Limitations and Future Work
The authors acknowledge several limitations:
- Dataset Issues: Some texts were not clearly news reports but rather personal social media posts, and some Hindi texts were short and unintelligible.
- Labeling Ambiguity: The labels "bias against Palestine" and "bias against Israel" are considered problematic, as bias can exist towards Israel without necessarily being against Palestine, and vice versa. They suggest renaming these to "bias pro-Israel" and "bias pro-Palestine" for better clarity.
- Representational Issues: The labels "bias against Palestine" and "bias against Israel" might not fully represent the entire countries, as entities like Hamas or the Israeli government don't always represent their entire populations.
- Frame Annotation Incompleteness: The frame annotation layer was not fully completed due to the deadline, and the authors plan to refine this scheme and release complete annotations in future work.
Future work includes expanding the set of frames, annotating more texts, applying the scheme to other datasets, and developing computational models for frame detection and analyzing the relationship between frames and bias.
Summary updated