The BLCU System in the BEA 2019 Shared Task

Liner Yang, Chencheng WangACL 2019

TL;DR

The BLCU team tackled the BEA 2019 Shared Task on Grammatical Error Correction (GEC) by developing a Transformer-based system that leverages data augmentation and an ensemble of models. Their approach achieved impressive results, ranking twelfth and fourth in the Restricted and Unrestricted tracks, respectively, demonstrating the effectiveness of their methods in improving GEC performance.

What Problem Does This Paper Address?

Grammatical Error Correction (GEC) is a crucial task in Natural Language Processing, aiming to automatically detect and correct grammatical errors in text written by non-native English speakers. While previous GEC shared tasks have provided valuable benchmarks, the BEA 2019 Shared Task introduced a new challenge by providing datasets with varying CEFR levels, requiring systems to be robust across different language proficiencies. The task also presented a need for effective utilization of both provided learner corpora and potentially external monolingual data to achieve high performance.

Methods

The BLCU system is built upon the Transformer architecture, a powerful neural network model that has become the de facto standard for many NLP tasks, including GEC. Their approach incorporates several key techniques:

  • Data Augmentation: A significant contribution is their method of corrupting a large monolingual corpus (One Billion Word Benchmark) to create synthetic training data. This process involves randomly deleting, adding, or replacing tokens based on predefined probabilities, mimicking common error types. This augmentation is crucial for improving the model's ability to handle errors, especially in the Restricted Track where external data is limited.
  • Byte Pair Encoding (BPE): To address the issue of large vocabularies and out-of-vocabulary (OOV) words, the authors employ BPE for subword tokenization. This allows the model to handle rare words and morphological variations more effectively.
  • Language Model-based Spell Checker: Recognizing that Transformer models might not be inherently optimized for spelling correction, they integrate a KenLM-based 5-gram language model with CyHunspell to specifically address spelling errors. This acts as a post-processing step to refine the output.
  • Model Ensemble: To further boost performance, the BLCU team ensembles eight single Transformer models. These models are trained with the same architecture and hyperparameters but differ in their random seeds, a common practice to capture diverse learning patterns.
  • Pre-training and Fine-tuning: The training strategy involves pre-training the Transformer model on the corrupted monolingual data for several epochs, followed by fine-tuning on the provided learner corpora. This two-stage approach allows the model to first learn general error patterns from augmented data and then adapt to the specific characteristics of learner English.

Datasets

The BLCU system was trained and evaluated on several datasets:

  • Learner Corpora (for Track 1 - Restricted):
    • NUCLE
    • Lang-8
    • FCE
    • W&I+L (Write & Improve + LOCNESS)
  • Monolingual Corpus (for Data Augmentation):
    • One Billion Word Benchmark dataset
  • Additional Corpus (for Track 2 - Unrestricted):
    • Non-public Lang-8 corpus

The BEA 2019 Shared Task provided these corpora with different CEFR levels (A, B, C) and also included essays written by native English students (N) for evaluation.

Key Insights and Ideas

  • Data Augmentation is Key: The paper strongly suggests that augmenting training data by corrupting monolingual corpora is a highly effective strategy for improving GEC performance, especially when learner-specific data is limited.
  • Ensembling for Robustness: Combining multiple Transformer models, even those trained with the same architecture, leads to a significant performance boost, indicating that diverse learning paths contribute to better generalization.
  • Hybrid Approach Pays Off: The integration of a dedicated spell checker alongside the Transformer model addresses a potential weakness of pure NMT-based GEC systems, leading to more comprehensive error correction.
  • CEFR Level Impact: The results suggest that performance can vary across different CEFR levels, with potential overtraining on beginner errors impacting performance on more advanced levels (C and N).

Key Results

The BLCU system achieved competitive results in the BEA 2019 Shared Task:

  • Restricted Track: F0.5 score of 58.62, ranking twelfth.
  • Unrestricted Track: F0.5 score of 59.50, ranking fourth.

The paper highlights that the data augmentation method contributed significantly to performance, yielding an increase of about 3.06 in F-measure on a single model. The spell checker further improved performance by about one point. The use of the Non-public Lang-8 corpus in the Unrestricted Track significantly improved recall by 5-10 points, though it led to a slight drop in F0.5 for higher CEFR levels.

Limitations and Future Work

  • CEFR Level Performance Discrepancy: The authors note that overtraining on data with beginner errors might have reduced performance on higher CEFR levels (C and N). This suggests a need for more nuanced training strategies that can better balance performance across different proficiency levels.
  • Model Diversity: While they ensembled identical Transformer models, they suggest that exploring different types of models in future work could further enhance performance.
  • Domain Mismatch: The corrupted corpus and learner corpus belong to different domains, and the errors in the augmented data are not always common for second language learners. While they trained separately, further research into bridging this domain gap could be beneficial.

Summary updated

BibTeX
@inproceedings{yang2019blcu,
  author = {Liner Yang and Chencheng Wang},
  title = {{The BLCU System in the BEA 2019 Shared Task}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure