TL;DR
This paper details a team's participation in the FIGNEWS 2024 shared task on annotating bias and propaganda in Gaza conflict news. They developed detailed annotation guidelines emphasizing annotator awareness of their own biases and explored integrating LLMs like ChatGPT to improve consistency, achieving moderate inter-annotator agreement.
What Problem Does This Paper Address?
Identifying and mitigating bias in news media is crucial for fostering an informed public. However, this field faces significant challenges: data scarcity, a lack of standardized definitions for bias and annotation, and generally low inter-annotator agreement (IAA). Existing large-scale datasets are often limited in size and may rely on non-expert annotators. The FIGNEWS 2024 shared task specifically aimed to address these issues by focusing on annotating bias and propaganda in news related to the Gaza conflict, with a particular emphasis on developing robust annotation guidelines.
Methods
The team employed a lightweight, iterative approach to guideline development, starting with literature research and trial annotations. They refined these guidelines based on annotating a substantial dataset, incorporating detailed examples and edge cases. A key methodological innovation was the emphasis on "conscious bias methodology," encouraging annotators to reflect on and discuss their own biases to mitigate their impact. They also explored integrating ChatGPT as an annotator, using specific prompts to ensure context-specific analysis and aiming to improve annotation quality through a two-way system for review and arbitration.
Datasets
The primary dataset used was from the FIGNEWS-2024 shared task, focusing on news related to the Gaza conflict. This dataset was split into two sections:
- Main dataset: 1800 texts for annotation.
- IAA dataset: 200 texts from the same batches, independently annotated by all participants to measure inter-annotator agreement.
Key Insights and Ideas
- Bias is inherent: The paper highlights that bias is not only present in human annotators but also in AI models like ChatGPT, underscoring the need for diverse annotator pools to achieve genuine consensus rather than homogeneity.
- Annotator awareness is key: Explicitly identifying bias and propaganda indicators, and encouraging annotators to reflect on their own biases, significantly aids in making more informed decisions.
- LLMs can assist, but require careful prompting: ChatGPT showed potential in assisting human annotators by identifying potential bias-carrying words and improving annotation quality, but its performance was sensitive to prompt design and required data cleaning to address misinterpretations.
- Subjectivity is a persistent challenge: Even with expert annotators and detailed guidelines, bias annotation remains highly subjective, leading to moderate IAA.
- Context and translation are tricky: The absence of full article context and the potential for connotation shifts in translated texts pose significant challenges for accurate bias detection.
Key Results
- The team developed detailed guidelines for text-level annotation of bias and propaganda.
- They annotated 2,000 texts in total.
- Achieved Krippendorff's alpha agreement of 0.43 for bias annotation and 0.30 for propaganda annotation, indicating fair to moderate agreement.
- After cleaning ChatGPT's annotations, the Krippendorff's alpha for bias improved to 0.315, comparable to human annotator agreement. For propaganda, it was 0.346.
- The label distribution showed a significant portion of texts categorized as "Unbiased" (31.6%) and "Biased against others" (27.3%), with smaller percentages for "Biased against Israel" (10.7%) and "Biased against Palestine" (2.6%). A substantial portion was also labeled "Unclear" (21.4%).
Limitations and Future Work
- Limitations:
- The approximately one-month time frame limited the extent of training and alignment opportunities for annotators.
- The submission deadline was a limiting factor in achieving higher IAA.
- The inherent subjectivity of bias annotation makes achieving high IAA challenging.
- Issues with data context and translated texts were noted.
- Future Work:
- Introduce labels for positive bias to provide a more comprehensive analysis.
- Split the labeling process into distinct steps (bias detection, polarity tagging, target identification) to enhance accuracy.
- Implement multi-layered labeling to account for different voices within texts (e.g., reporter vs. quoted sources).
- Advocate for N-way annotation for all examples to improve accuracy and identify confusing tags.
- Further explore the use of LLMs for bias annotation with improved prompting and integration strategies.
- Test bias annotation guidelines with a diverse pool of annotators to ensure high IAA reflects quality guidelines, not just annotator homogeneity.
Summary updated