TL;DR
This paper details the creation and application of data annotation guidelines for propaganda detection in news items, developed by the University of Tripoli team for the FIGNEWS 2024 shared task. Their rigorous annotation process, which involved iterative refinement and collaborative discussions, resulted in a dataset that helped them achieve third place in the competition's consistency track.
What Problem Does This Paper Address?
In today's digital landscape, the rapid spread of propaganda through news and social media poses a significant challenge. Identifying propaganda is complex, as it often blurs the lines with persuasive techniques. This paper addresses the critical need for high-quality, consistently annotated datasets to train and evaluate AI models for propaganda detection. The authors highlight that while AI research has made strides, the availability of reliable training data is crucial, and existing annotation guidelines may not always be sufficient for complex, nuanced tasks like propaganda identification.
Methods
The core of this paper lies in the development of comprehensive data annotation guidelines. The team at the University of Tripoli, comprising five members with backgrounds in IT and computer science, worked under the supervision of two senior experts. Their methodology involved:
- Iterative Guideline Development: Starting with the shared task's defined categories, the team developed heuristic guidelines, refining them through frequent group discussions and iterations to ensure clarity and consistency.
- Experimental Dataset Usage: An initial set of 100 posts was used for familiarization and initial discussions. ChatGPT-generated datasets were also employed during the guideline writing phase.
- In-house Annotation Tool: To streamline the process and allow for simultaneous work by multiple annotators, an in-house tool was developed.
- Training and Calibration: A thorough training session was conducted to ensure all annotators understood the guidelines. Regular calibration sessions and discussions were held to address ambiguities and ensure a shared understanding.
- Inter-Annotator Agreement (IAA): The team actively monitored and worked to improve IAA, a key metric for annotation quality and consistency. Disputes were resolved through team leader-led discussions.
Datasets
The primary dataset used was provided by the FIGNEWS 2024 shared task organizers. The University of Tripoli team annotated a total of 2600 news articles for this dataset. They also utilized ChatGPT-generated datasets during the guideline development phase.
Key Insights and Ideas
- The Importance of Clear, Unambiguous Criteria: The paper strongly emphasizes that precise and unambiguous criteria are essential for efficient and consistent annotation, especially for subjective tasks like propaganda detection.
- Collaborative Annotation is Key: The success of the team in achieving high consistency was attributed to collaborative efforts, regular discussions, and calibration sessions, which fostered a shared understanding of the guidelines.
- Iterative Refinement Yields Better Guidelines: The process of designing guidelines was not a one-off event but an iterative one, involving feedback and adjustments, which ultimately led to more effective criteria.
- AI's Role in Facilitating Annotation: The development of an in-house annotation tool demonstrates how AI-related tools can significantly speed up and improve the annotation process.
- Subjectivity and Ambiguity are Inherent Challenges: The authors acknowledge that the ambiguity and subjectivity involved in defining propaganda are significant barriers that require careful handling.
Key Results
The University of Tripoli team achieved third place among six participating teams in the consistency track of the FIGNEWS 2024 shared task. Their Kappa score for IAA quality was 18.7. For the Guidelines track, they satisfied all requirements and scored 7 out of 8. They successfully annotated 2600 data points, meeting the shared task requirements for the Quantity track.
Limitations and Future Work
The authors acknowledge several limitations:
- Time Constraints: The project was constrained by time limitations.
- Annotator Background: They suggest that including individuals with data science backgrounds, in addition to IT and computer science, could have yielded even better results.
- Annotator Diversity: A more varied set of annotators could further enhance the guidelines.
Future work includes:
- Refining Guidelines: Further refining the guidelines based on feedback from the broader research community.
- Automated Annotation Tools: Incorporating these guidelines into automated annotation tools to optimize the process.
- Advancing Propaganda Study: The authors believe their work will contribute to the systematic study of news media propaganda.
Summary updated