TL;DR
This paper introduces DRAGON, a novel Retrieval Augmented Generation (RAG) approach for detecting bias and propaganda in social media news related to the Gaza-Israel conflict. By leveraging topic modeling to create a representative data subset and then employing a voting mechanism of multiple classifiers (including RAG), they achieve robust bias and propaganda detection.
What Problem Does This Paper Address?
The paper tackles the challenge of accurately identifying bias and propaganda in news articles, particularly in the context of sensitive geopolitical events like the Gaza-Israel conflict. Existing methods often struggle with the sheer volume and nuanced nature of social media discourse. The FIGNEWS Shared task aims to create a comprehensive corpus for analyzing bias and double standards across diverse perspectives and languages. This work contributes to this effort by proposing an efficient and effective annotation and classification methodology.
Methods
The core of the proposed method, DRAGON, involves a multi-stage process:
- Topic Modeling for Data Subset Selection: They use BERTopic to identify 35 distinct topics within the initial 13,500-sample FIGNEWS dataset. From each topic, approximately 50 representative samples are selected, creating a more manageable and representative subset of 1,812 samples for manual annotation. This is a key innovation for efficient data curation.
- Manual Annotation: The selected subset is meticulously annotated for bias and propaganda labels by human annotators.
- Multi-Classifier Ensemble with Voting: Four classification models (KNN, SVC, XGBoost, and a RAG-based model) are trained on the annotated subset.
- RAG for Enhanced Annotation: The RAG component is particularly interesting. It integrates label definitions directly into LLM prompts and retrieves similar annotated examples from the curated subset to provide richer context for classification. This dynamic prompting approach aims to improve accuracy.
- Voting Mechanism: For annotating the remaining dataset, a voting system combines the predictions of the four trained classifiers. In cases of ties, a criterion based on F1 scores is used to select the final label.
Datasets
- FIGNEWS Shared-task dataset: An initial dataset of 13,500 social media news samples related to the Gaza-Israel conflict.
- Annotated Subset: A curated subset of 1,812 samples, selected using topic modeling and then manually annotated for bias and propaganda.
Key Insights and Ideas
- Topic Modeling for Efficient Annotation: Using topic modeling (BERTopic) to select a representative subset of data significantly streamlines the annotation process and ensures coverage of diverse themes within the dataset.
- RAG's Contextual Power: Integrating label definitions and similar examples directly into LLM prompts via RAG proves to be a powerful technique for enhancing classification accuracy, especially for nuanced labels.
- Ensemble Learning for Robustness: Combining predictions from multiple diverse classifiers (KNN, SVC, XGBoost, RAG) through a voting mechanism leads to more robust and reliable predictions than any single model alone.
- Annotator Bias is a Real Challenge: The paper highlights that even with clear guidelines, cultural and personal beliefs of annotators (in this case, from Iran and the USA) can introduce biases and inconsistencies, underscoring the need for diverse annotation teams and robust bias mitigation strategies.
- Unbalanced Data Impacts Performance: The low frequency of certain labels (e.g., "Biased against both Palestine and Israel") negatively affects the performance of automated annotators, emphasizing the importance of addressing data imbalance.
Key Results
- The paper demonstrates the effectiveness of their topic-modeling-driven subset selection and multi-classifier ensemble approach for bias and propaganda detection.
- While specific F1 scores for the final DRAGON model on the full dataset are not explicitly detailed, the paper presents F1 scores for individual classifiers and embedding models in Tables 5 and 6, showing competitive performance. For instance, in Table 6 (Bias results), KNN and SVC with the Multilingual E5 large embedding achieve a weighted F1 score of 0.78.
- The RAG method, when integrated with dynamic prompts and similar examples, shows promise for improving classification.
Limitations and Future Work
- Limited Human Annotators: The reliance on only two human annotators from different countries introduced biases and inconsistencies, suggesting a need for a larger and more diverse annotation pool.
- Data Imbalance: The low frequency of some labels poses a challenge for model performance, indicating a need for techniques to address data imbalance.
- RAG Improvement: The authors acknowledge that their RAG solution has potential for further improvement, especially given the rapid advancements in LLMs and retrievers. They suggest that RAG-based solutions can offer better generalization and reduce the impact of data biases.
- Larger Annotated Subsets: The findings suggest that increasing the size of the manually annotated subset could further enhance the performance of automated classification systems.
Summary updated