TL;DR
This paper presents the CUNI system for the BEA 2019 Shared Task on Grammatical Error Correction (GEC). They achieved strong results across three tracks by leveraging Transformer models and introducing several enhancements, including novel dropout strategies, iterative decoding, and extensive data augmentation. Their system placed third in the Unrestricted Track, demonstrating the effectiveness of their approach.
What Problem Does This Paper Address?
Grammatical Error Correction (GEC) is a crucial task in natural language processing, aiming to automatically correct errors in text. While significant progress has been made, particularly with machine translation-based approaches, challenges remain, especially in low-resource settings and when dealing with diverse error types. The BEA 2019 Shared Task specifically aimed to push the boundaries of GEC across different scenarios: restricted (limited training data), unrestricted (any training data allowed), and low-resource (no annotated corpora). This paper addresses how to build high-performing GEC systems for these varied conditions, focusing on improving robustness and accuracy.
Methods
The CUNI system is built upon the Transformer architecture, a standard in modern NLP. They introduced several key improvements:
- Enhanced Dropout: Beyond standard dropout, they applied dropout to entire source and target words (embedding vectors), aiming to prevent over-reliance on specific words.
- Weighted Target Subwords: They experimented with weighting target subwords based on their alignment to source words, potentially improving translation quality.
- Model Checkpoint Averaging: Averaging the last few model checkpoints during training was found to reduce variance and improve performance, a common technique for stabilizing training.
- Iterative Decoding: To address the issue of models converging to local optima and failing to correct all errors, they employed iterative decoding. This process repeatedly applies the GEC model to its own output, allowing for incremental corrections as long as the corrected sentence is deemed more probable.
- Data Augmentation and Oversampling: For the Restricted Track, they significantly oversampled existing datasets to better match the target domain. For the Low-Resource Track, they utilized a massive dataset of Wikipedia revision histories, which, while noisy, provided a large source of weakly supervised data.
- Edited Maximum Likelihood Estimation (MLE): They modified the standard MLE objective to assign higher weights to target tokens that differ from source tokens, encouraging more substantial corrections.
Datasets
The paper utilized several datasets for training and evaluation:
- Restricted Track: FCE v2.1, Lang-8 Corpus of Learner English, NUCLE, Write & Improve (W&I), and LOCNESS v2.1.
- Low-Resource Track: Wikipedia revision dumps (approximately 190 million segment pairs). This dataset is characterized by its large size but also its inherent noise and lack of direct GEC annotation.
- Development Sets: Concatenated W&I and LOCNESS development sets were used for ablation studies.
Key Insights and Ideas
- Wikipedia as a Weakly Supervised GEC Source: The success in the Low-Resource Track highlights the potential of leveraging large, noisy, automatically extracted datasets like Wikipedia edits for GEC, even when they aren't perfectly curated for the task.
- Iterative Decoding's Trade-off: Iterative decoding, while improving recall by allowing more corrections, can lead to a decrease in precision. Finding the right balance through threshold tuning is crucial.
- Data Quality vs. Quantity: While the Wikipedia data was massive, it was also noisy. The paper shows that even noisy data can be beneficial, especially when combined with effective training strategies.
- Regularization's Impact: In the Low-Resource Track, the
transformer_clean_big_tpuconfiguration (without internal dropouts) performed best, suggesting that with a sufficiently large dataset, explicit regularization might not always be beneficial and could even be detrimental.
Key Results
- Restricted Track: Achieved an F0.5 score of 59.39, ranking 10th out of 21 participants.
- Low-Resource Track: Achieved an F0.5 score of 44.13.
- Unrestricted Track: Achieved an F0.5 score of 64.55, ranking 3rd out of 7 submissions. This was achieved by finetuning the Low-Resource model on the annotated data.
Limitations and Future Work
- Iterative Decoding Threshold: The optimal threshold for iterative decoding can be sensitive and might benefit from dynamic adjustment within the decoding process itself.
- Spelling/Infillment Noise Evaluation: The paper notes that it would be interesting to evaluate the effect of spelling and infillment noise separately in the Low-Resource Track.
- Cross-lingual GEC: The authors suggest exploring Wikipedia revisions in other languages as a future direction for GEC research.
Summary updated