TL;DR
This paper introduces ALEN, a novel app that provides English language learners with automated, discourse-level feedback on their argumentative writing. By leveraging argumentation mining, ALEN helps students identify and repair broken claim-premise structures, aiming to improve both their writing skills and their overall English language learning. Early evaluations suggest ALEN positively impacts student self-efficacy and technology acceptance.
What Problem Does This Paper Address?
Learning to write persuasively is a crucial skill, especially for English language learners (ELLs). While Natural Language Processing (NLP) and Machine Learning (ML) have been integrated into educational tools for tasks like grammar checking and vocabulary building, there's a significant gap in supporting the discourse level of argumentation. Specifically, tools that help learners understand and construct logical arguments—identifying claims and the premises that support them—are scarce. This paper addresses the challenge of providing effective, automated feedback on argumentation structure to foster "arguing to learn," a process where argumentation itself aids in acquiring other knowledge, in this case, English language proficiency.
Methods
The core of the ALEN app is an argumentation mining model trained to detect claim-premise structures and their relationships within student essays. This model was developed using a Support Vector Machine (SVM) classifier, achieving accuracies of 65.4% for argument component classification and 72.1% for persuasive relation classification.
The ALEN app itself is designed as a user-friendly writing support tool. It features a simple text input field where students can write or paste their essays. Upon analysis, ALEN provides a "personal argumentation dashboard" offering:
- Visual Feedback: Claims are highlighted in green, and premises in yellow within the text.
- Graph-based Representation: A visual overview of the text's argumentation structure.
- Summarizing Scores: Metrics for readability, coherence, and persuasiveness.
- Detailed Explanations: "Explanation" or "Help" buttons provide insights into argumentation theory and best practices.
The methodology for building ALEN involved three steps: literature analysis, corpus identification and modeling, and implementation of the adaptive writing support system.
Datasets
The argumentation mining model was trained on a corpus of 1000 business model peer feedback essays written by students. This corpus, published by Wambsganss et al. (2020b), was chosen because it contained annotated persuasive student essays, was sufficiently large for real-world scenarios, and used a rigorous annotation guideline.
For evaluation, students were asked to complete a Cambridge English Assessment task for the B2 language level.
Key Insights and Ideas
- Bridging "Arguing to Learn" and NLP: The paper highlights the under-explored potential of NLP-based argumentation feedback for "arguing to learn," where the act of constructing arguments enhances learning itself.
- Visualizing Argument Structure: The use of in-text highlighting and graph-based representations is identified as an effective way to guide learners in understanding their argumentation.
- Adaptive Feedback for ELLs: The ALEN app demonstrates how adaptive, discourse-level feedback can be tailored to support the specific needs of English language learners.
- Student Perceptions Matter: The study emphasizes the importance of user perception, showing that students found the tool easy to use, useful, and intended to use it further.
Key Results
The paper reports on two field studies involving 28 German high school students.
Study 1 (10 participants): * Perceived ease-of-use: Mean score of 5.77 (normalized 0.82) * Perceived usefulness: Mean score of 5.60 (normalized 0.8) * Intention to use: Mean score of 5.7 (normalized 0.81) These scores, all above the midpoint of 4, indicate positive technology acceptance.
Study 2 (18 participants): * Self-efficacy for English language learning tasks: Mean score of 5.02 (normalized 0.71). This suggests ALEN might increase engagement and motivation.
Qualitative Findings: * Students favorably noted the adaptive feedback, in-text highlighting, and graph overview. * Some students found that persuasive elements were sometimes wrongly highlighted. * A common request was for additional grammar feedback, as it was sometimes unclear if an argument was weak or if the grammar was simply erroneous.
Limitations and Future Work
Limitations: * The study was a small-scale proof-of-concept with only 28 participants. * The dataset used for training was specific to business model peer feedback, which might limit generalizability to other domains. * Students sometimes struggled to differentiate between argumentation errors and grammatical errors.
Future Work: * Combine Discourse and Grammar Feedback: Integrate grammar feedback with discourse-level argumentation feedback for more comprehensive guidance. * Investigate Human-Computer Interaction: Further research into the interaction design of discourse-level writing support tools for language learners. * Large-Scale Field Studies: Conduct larger studies to measure the long-term learning success and impact on students' language learning outcomes.
Summary updated