TL;DR
This paper introduces a novel annotation scheme and a freely available corpus of 1,000 German-language persuasive peer reviews. This resource aims to advance the development of AI-powered writing support systems that can provide students with tailored feedback on their argumentation skills, addressing a critical need in modern education.
What Problem Does This Paper Address?
In today's educational landscape, there's a growing emphasis on higher-order thinking skills like critical thinking and argumentation. However, providing personalized, formative feedback on these skills is challenging, especially in large classes or online learning environments like MOOCs. While Argumentation Mining (AM) offers a promising avenue for automated feedback, there's a significant lack of suitable linguistic corpora, particularly in German and within specific educational contexts like peer reviews. Existing corpora are often limited in size, language, or the depth of their argumentation annotations, making them insufficient for training robust writing support systems. This paper aims to bridge this gap by creating a German-language corpus of student peer reviews annotated for argumentation components and their relations.
Methods
The authors developed a new annotation scheme for modeling argumentation discourse structures in persuasive peer reviews. This scheme is based on established argumentation theory (Toulmin, 1984) and prior work on annotating persuasive essays. The annotation process involved:
- Literature Review: Examining existing theories and annotation schemes for argumentation.
- Guideline Development: Creating a 15-page set of annotation guidelines based on a random sample of 50 peer reviews.
- Pre-study and Refinement: Conducting three workshops with three native German speakers to resolve ambiguities and refine the guidelines.
- Annotation: Applying the finalized guidelines to annotate a corpus of 1,000 student-generated peer reviews using the brat rapid annotation tool.
The annotation focused on identifying "claims" (the central arguable unit) and "premises" (statements supporting or attacking the claim), as well as "support" and "attack" relations between them.
Datasets
The paper introduces a new dataset:
- German Argumentative Student Peer Reviews Corpus: A collection of 1,000 persuasive peer reviews written in German by students in a business innovation lecture. This corpus is annotated for argument components (claims and premises) and argumentative relations (support/attack). It is made freely available to the research community.
The authors also reference existing corpora like the Araucaria corpus, ECHR corpus, Debatepedia corpus, and corpora by Stab and Gurevych, highlighting their limitations for the specific educational context and language addressed in this paper.
Key Insights and Ideas
- Peer Reviews as a Rich Source: Student peer reviews, particularly in contexts like business model development, naturally contain argumentative feedback, making them an ideal domain for studying and supporting argumentation skills.
- Annotation Reliability is Achievable: Despite the inherent subjectivity in identifying argumentative structures, the study demonstrates that with clear guidelines and training, annotators can achieve moderate to substantial agreement on claims, premises, and relations.
- Claims vs. Premises Can Be Tricky: The analysis revealed that distinguishing between claims and premises can be challenging, especially in reasoning chains where a statement can function as a claim in one context and a premise in another. This highlights the importance of context and authorial intent.
- Support Relations Dominate: In this specific domain of peer reviews, "support" relations are overwhelmingly more common than "attack" relations, reflecting the constructive nature of feedback (highlighting strengths, weaknesses, and suggestions).
Key Results
- Inter-Annotator Agreement (IAA):
- Argument Components: Achieved moderate agreement (e.g., multi-π scores of 55.47% for claims and 51.06% for premises, Krippendorff's α around 55% for claims and 51% for premises).
- Argumentative Relations: Achieved high agreement for support/non-support relations (IAA of 94.13%, multi-π and Krippendorff's α of 49.03%).
- Corpus Size: The final corpus contains 20,125 sentences and 246,980 tokens.
- Annotation Distribution: Approximately 31.64% of text spans were identified as claims, 33.55% as premises, and 34.81% as non-argumentative.
- Machine Learning Performance: A Support Vector Machine (SVM) achieved 65.4% accuracy for argument component classification and 72.1% accuracy for argumentative relation classification on the test set.
Limitations and Future Work
- Limited Component Granularity: The current annotation scheme does not distinguish finer-grained types of claims (e.g., strength, weakness, recommendation) or premises (e.g., example, statistic, intuition). The authors suggest this would be a useful addition for future work.
- Low Attack Relation Frequency: The scarcity of "attack" relations in the corpus limits the ability to train models specifically for detecting and analyzing argumentative attacks.
- Boundary Detection: While classification agreement was moderate, the detection of precise boundaries for argument components showed slightly lower agreement.
- Future Work: The authors hope their corpus and annotation scheme will encourage further research in designing and developing argumentative writing support systems, particularly for large-scale learning scenarios. They also mention the potential for using this corpus to train models for automated essay scoring and to explore more nuanced argumentation structures.
Summary updated