TL;DR
This paper introduces a novel approach for automatically detecting and annotating media bias in news narratives concerning the Israel-Gaza conflict. By leveraging Large Language Models (LLMs) and sophisticated prompt engineering, the authors achieve high accuracy in classifying biased content, offering a valuable tool for researchers and a step towards more objective media analysis.
What Problem Does This Paper Address?
In today's digital age, social media and online news platforms are primary sources of information, often perceived as highly reliable. However, these sources can be rife with bias, subtly shaping public opinion, especially during sensitive geopolitical events like the Israel-Gaza war. Manually identifying and annotating this bias is a time-consuming and subjective process. The FIGNEWS 2024 shared task aims to address this by creating a benchmark for automated bias detection in news media narratives. This paper tackles the challenge of developing an automated system that can accurately and consistently identify various forms of media bias within these narratives, moving beyond manual annotation.
Methods
The core of this work lies in the application of Large Language Models (LLMs), specifically OpenAI's GPT-3.5 Turbo API, for bias detection. The authors employed a prompt engineering technique, which involves carefully crafting detailed instructions and examples to guide the LLM. This process was iterative:
- Initial Prompting: The LLM was provided with definitions of bias categories and general instructions.
- Contextual Refinement: Early experiments revealed that the model struggled with context-specific terms (e.g., Hamas, IDF). To mitigate this, the prompt was enhanced with explicit definitions for key entities and terms relevant to the conflict.
- Example-Based Learning: The prompt was further enriched with numerous examples for each bias category, demonstrating how to classify similar statements. This served as in-context learning for the LLM.
- Iterative Improvement: The prompt was continuously refined based on validation and testing, incorporating additional context and addressing identified ambiguities and errors.
- Ethical Considerations: To counter potential inherent biases in the LLM, the model was instructed to focus solely on provided context, and its "temperature" parameter was set low (0.4) to minimize hallucinations.
Datasets
The study utilized data from the FIGNEWS 2024 shared task, which comprises news headlines, articles, and social media posts related to the Israel-Gaza war (2023-2024). The authors manually labeled a portion of this dataset to serve as training data and for validation. While the paper doesn't explicitly state the total size of the FIGNEWS dataset, it mentions that their manual labeling efforts involved 100 statements for initial training and 40% of the manually labeled data for validation.
Key Insights and Ideas
- LLMs as Powerful Bias Detectors: The research demonstrates that LLMs, when guided by well-crafted prompts and examples, can be highly effective tools for automated bias detection, rivaling human annotator capabilities.
- The Importance of Contextual Definitions: The initial struggles of the LLM highlight that simply providing bias categories is insufficient. Explicitly defining key entities and terms within the prompt is crucial for accurate interpretation and classification.
- Bias is Pervasive: The analysis of the labeled data revealed a significant prevalence of biased content, with a substantial portion of statements leaning against Palestine. This underscores the critical need for tools that can help identify and understand this bias.
- Nuances in Bias: The word cloud analysis provides a qualitative insight into how bias manifests, showing specific terms and themes associated with different bias categories (e.g., "Hamas" and "terrorist" in bias against Palestine).
Key Results
The paper reports a 99% accuracy for their automated bias detection model. This high accuracy was achieved after an iterative refinement process of the LLM prompt. The manual annotation process involved three team members, and their inter-annotator agreement (IAA) score was 0.256 (Fleiss Kappa), indicating a reasonable level of agreement, which they aimed to match or exceed with their automated system.
Limitations and Future Work
The authors acknowledge several limitations:
- Time Consumption: The LLM's annotation process was time-consuming, taking approximately 1 hour for 500 statements.
- Closed-Source Nature: The reliance on a closed-source LLM (GPT-3.5 Turbo API) incurred costs and limited the ability to further modify the model's internal workings.
- Limited Examples: The number of manually labeled examples was constrained by time and cost, which could have further improved IAA and model accuracy.
Future work suggested includes improving the IAA score, further enhancing the prompt to achieve even better accuracy, and potentially exploring more efficient annotation methods.
Summary updated