TL;DR
This paper explores the feasibility of using Walton's extensive set of argumentation schemes for annotating arguments in Swedish political texts. The study reveals significant challenges in achieving high inter-annotator agreement, highlighting the need for more refined instructions and potentially a simplified scheme for practical argumentation mining.
What Problem Does This Paper Address?
Argumentation mining, the automatic identification and classification of arguments within text, is a valuable tool for various applications. A crucial aspect of developing robust argumentation mining systems is the creation of high-quality annotated datasets, often referred to as "gold standards." These datasets are essential for training and evaluating machine learning models. However, creating such datasets requires an annotation scheme that is both expressive enough for the task and clear enough for human annotators to apply consistently and accurately. The challenge lies in finding or developing annotation schemes that achieve high inter-annotator agreement, meaning different annotators consistently agree on their annotations. This paper investigates the suitability of a comprehensive set of argumentation schemes developed by Walton et al. (2008) for this purpose, specifically within the context of Swedish political texts.
Methods
The study involved two linguistically trained annotators who manually annotated a dataset of Swedish newspaper editorials using Walton's argumentation schemes. The annotation process was guided by the descriptions provided in Walton et al. (2008), with minimal explicit instruction beyond an initial practice session and a follow-up discussion. Annotators used the Araucaria tool, which supports Walton's schemes, to identify argument units (premises and conclusions) and assign them to specific schemes. The paper then analyzes the agreement between the two annotators, focusing on the number of annotated arguments, the identification of premises and conclusions, and the classification of argumentation schemes. They also explore the impact of grouping schemes to improve agreement.
Datasets
The dataset used in this study consists of 30 Swedish newspaper editorials, originally compiled by Hedquist (1978) for research on emotive language. These editorials, totaling approximately 19,000 words, were selected from six newspapers published in May-September 1973, representing the five political parties in the Swedish parliament at the time. The dataset was not pre-annotated for argumentation structure.
Key Insights and Ideas
- Walton's schemes are complex for annotation: Despite the detailed descriptions, achieving high inter-annotator agreement with Walton's extensive set of schemes proved challenging for the annotators.
- Disagreement on argument structure: Annotators differed not only in the number of arguments they identified but also in how they segmented arguments into premises and conclusions.
- Scheme ambiguity: The specific labels of the argumentation schemes were a significant source of disagreement, suggesting that some schemes are difficult to distinguish from one another in practice.
- Grouping schemes improves agreement: Collapsing the detailed schemes into broader categories, as suggested by Walton et al. (2008), led to improved agreement among annotators.
- Implication for annotation guidelines: The findings highlight the critical need for clear, detailed, and potentially simplified annotation guidelines to ensure consistency and accuracy in argumentation mining.
Key Results
The study found significant differences between the two annotators in several aspects:
- Number of arguments: Annotator 1 (A1) annotated approximately 150 more argument instances than Annotator 2 (A2).
- Premise/Conclusion identification: A2 annotated more premises on average, and annotators differed in how they handled missing premises and conclusions, with A1 adding significantly more "missing" units.
- Scheme usage: A1 used 12 schemes, while A2 used 21, with a total of 22 different schemes used across both annotators. Despite A2 using more schemes, A1 annotated more argument instances.
- Scheme agreement: When comparing arguments with matching conclusions and premises, the agreement on the specific scheme was very low. For example, with a strict overlap threshold (α=0.9), only 2 arguments had matching schemes.
- Grouped schemes show better agreement: When schemes were grouped, inter-annotator agreement improved, though it remained relatively low for full argument matches.
Limitations and Future Work
The authors acknowledge several limitations and propose future directions:
- Limited annotator pool: The study used only two annotators, which makes it difficult to generalize findings about "normal" disagreement.
- Instructional clarity: The annotators received minimal explicit instruction, suggesting that more detailed and precise instructions could improve agreement. However, this might also lead to a loss of information.
- Scheme complexity: The inherent difficulty in distinguishing between some of Walton's schemes is a significant hurdle.
- Annotation workflow: The authors suggest that structuring the annotation task differently, perhaps by first identifying argument spans and then labeling components and schemes, could be beneficial.
Future work will involve designing further annotation exercises with:
- More explicit annotation manuals.
- Potentially less fine-grained schemes.
- Multiple annotation workflows (e.g., combined span identification and labeling vs. separate steps).
- A larger number of annotators to ensure more robust agreement analysis.
- Potentially extending the dataset.
Summary updated