TL;DR
This paper introduces LinggleWrite, a novel web-based system designed to act as an AI writing coach for essay composition. It goes beyond simple error correction by offering interactive suggestions, automated essay scoring, and detailed feedback, aiming to empower learners to improve their writing skills.
What Problem Does This Paper Address?
Essay writing is a crucial skill, often assessed in standardized tests, but providing effective, personalized feedback is challenging for educators due to heavy workloads. While existing computer-assisted writing tools can detect errors and assign scores, they often lack the "coaching" aspect – providing explanations, grammar patterns, and contextual examples that truly help learners improve. The paper addresses this gap by developing a system that not only identifies errors but also guides users toward better writing practices through interactive suggestions and detailed feedback.
Methods
LinggleWrite is built upon four core components:
- Interactive Writing Suggestion: As users type, the system provides real-time prompts related to grammar patterns, collocations, and bilingual examples. This is powered by extracting common grammar patterns from a large corpus.
- Essay Scoring (AES): The system assesses writing proficiency using CEFR levels (A1-C2). They formulate this as a regression problem, experimenting with both word-based (CNN, LSTM, Bi-LSTM) and sentence-based (LSTM-LSTM, LSTM-CNN) neural network architectures, enhanced with an attention mechanism.
- Grammatical Error Detection (GED): LinggleWrite detects grammatical errors by treating it as a sequence labeling problem. They introduce a novel "DIRC" (Delete, Insert, Replace, Correct) tag schema, which is more informative than binary (Correct/Incorrect) tags. Their GED model utilizes a BiLSTM-CRF architecture augmented with BERT and Flair embeddings for improved performance.
- Corrective Feedback: For detected errors, the system retrieves plausible corrections by querying the Linggle linguistic search engine. This involves designing specific queries based on the detected edit type (insertion, deletion, replacement) and using Linggle's operators to find relevant n-grams and bilingual examples.
Datasets
- For AES: The EF-Cambridge Open Language Database (EFCAMDAT) was used. This dataset contains approximately 1.2 million essays from around 174,000 learners across CEFR levels A1-C2. For training, they randomly selected 1,903 essays from each level and used 5-fold cross-validation.
- For GED: The First Certificate in English (FCE) dataset was used, containing 1,224 essays manually tagged with 77 error types. They used 30,953 sentences for training, 2,720 for testing, and 2,222 for development, converting the dataset into the DIRC format.
Key Insights and Ideas
- DIRC Tag Schema: The introduction of the DIRC tag schema for grammatical error detection is a significant contribution. It moves beyond simply identifying an error to specifying the type of correction needed (delete, insert, replace), providing more actionable feedback for learners.
- Integration of Multiple AI Components: The system effectively integrates AES, GED, and interactive suggestions, creating a more holistic writing assistance tool than many existing solutions.
- Leveraging Linguistic Search Engines: The use of Linggle to retrieve corrective feedback is a smart approach, allowing the system to tap into a vast linguistic resource for contextually relevant suggestions.
- Attention Mechanism in AES: The finding that sentence-level models with an attention mechanism outperform word-level models in essay scoring suggests that focusing on sentence structure and relationships is key to accurate assessment.
Key Results
- AES Performance: The LSTM-LSTM-ATT model achieved the highest average QWK score of 0.957 on the EFCAMDAT dataset, outperforming previous benchmarks. Sentence-level models generally performed better than word-level ones.
- GED Performance: The BiLSTM-CRF+BERT+Flair model achieved state-of-the-art performance on the FCE test set using the DIRC schema, demonstrating comparable results to binary models but with more informative feedback.
- Qualitative Feedback: The system provides interactive suggestions, collocations, and bilingual examples, aiming to guide users towards more fluent writing.
Limitations and Future Work
- Limitations: The paper acknowledges that automated grammatical error correction is still imprecise, which is why systems like Write&Improve only mark "highly-likely incorrect words." The performance of GED models, even state-of-the-art ones, is still below human annotator agreement levels, suggesting room for improvement.
- Future Work: The authors suggest several avenues for future research:
- Implementing methods for introducing additional training data or generating artificial data.
- Exploring re-ranking of corrective suggestions to prioritize the most relevant ones.
- Detecting finer-grained error types.
- Expanding language support for translations of examples.
Summary updated