Chinese Grammatical Error Correction Based on Convolutional Sequence to Sequence Model

Si Li, Jianbo Zhao, Guirong Shi, Yuanpeng Tan, Huifang Xu, Guang Chen, Haibo Lan, Zhiqing LinIEEE Access 2019 · DOI

Abstract

Grammatical error correction (GEC) for the Chinese language presents unique challenges due to its flexible structure and the localized nature of common errors. This paper introduces the first application of a convolutional sequence-to-sequence (ConvS2S) architecture to Chinese Grammatical Error Correction (CGEC). By leveraging the inherent ability of convolutional neural networks (CNNs) to capture local context and introducing optimizations such as shared embeddings and reinforcement learning via policy gradients, the researchers demonstrate significant performance gains over traditional recurrent neural network (RNN) approaches.

Introduction

Correcting grammatical errors in Chinese is notoriously difficult. Unlike English, where errors often involve discrete categories like articles or verb tenses, Chinese errors frequently involve misnomers—incorrect word choices based on similar pronunciation or form—and the misuse of auxiliary words. Furthermore, Chinese grammatical errors tend to be "clumpy," typically concentrated within spans of three to four words.

Traditional Sequence-to-Sequence (Seq2Seq) models based on RNNs process text linearly. This linear processing can be computationally expensive and may struggle to prioritize the tight, local clusters of errors characteristic of Chinese writing. This study investigates whether a convolution-based architecture, which naturally excels at capturing local patterns through fixed-size kernels, provides a superior fit for the specific "shape" of Chinese grammatical errors.

Methodology

The core of the proposed system is a Convolutional Sequence-to-Sequence (ConvS2S) architecture. Unlike RNNs, which rely on sequential processing where each step depends on the previous output, CNNs allow for parallelized computation. By stacking layers, the model can represent long-term dependencies while maintaining a focus on local features in the lower layers.

To enhance the model's performance, two primary optimizations were introduced:

1. Shared Embeddings

In standard machine translation, source and target vocabularies differ. However, in CGEC, both the input (erroneous sentence) and output (corrected sentence) are in Chinese. By forcing the encoder and decoder to share the same embedding weights, the model learns a more robust representation of word meanings. This reduces the total parameter count and ensures that a word's semantic vector remains consistent regardless of its position in a broken or corrected sentence.

2. Policy Gradient (Reinforcement Learning)

Because text generation is non-differentiable, standard optimization techniques can be limited. The authors employed a Monte-Carlo policy gradient method, using the GLEU score (a standard GEC metric) as a reward signal. This approach encourages the model to generate sentences that directly maximize the evaluation metric, rather than just minimizing cross-entropy loss.

Experimental Setup and Datasets

The study utilized a combination of established benchmarks and augmented data to train and evaluate the model:

  • Core Benchmarks: NLPCC 2018 and NLPTEA (2016-2018) provided error-annotated sentences.
  • Learner Corpora: Large-scale sentences from Chinese learners were pulled from HSK and Lang-8.
  • Data Augmentation: The researchers observed that original training sets often had a high ratio of "error" sentences to "correct" sentences (approximately 10:1). To prevent the model from over-correcting valid text, they integrated high-quality "correct" sentences from the Chinese Treebank (CTB9) and the Workshop on Machine Translation (WMT18).

Key Insights

The experimental results yielded several critical insights into the behavior of CGEC models:

  • The Local Context Advantage: The CNN’s sliding window (kernel) proved more efficient at identifying and fixing localized Chinese errors than the "long-memory" approach of RNNs.
  • Optimal Data Ratio: A significant finding was the impact of the error-to-correct sentence ratio. Training on a 6:1 ratio (error vs. correct) yielded the best results. A lower ratio led to over-correction, while a higher ratio diluted the model's ability to recognize error patterns.
  • Impact of Shared Embeddings: Implementing shared embeddings resulted in a 1.56% precision boost, confirming that semantic consistency is vital when the source and target languages are the same.
  • Precision vs. Recall Trade-off: While Policy Gradient optimization achieved the highest recall (identifying more errors), it resulted in a slight dip in precision. This suggests that reinforcement learning rewards like GLEU may require more granular, character-level constraints to avoid "fixing" grammatically correct segments.

Results and Discussion

The ConvS2S model demonstrated clear improvements over the RNN baseline. The basic convolutional model improved precision by over 6% compared to the RNN when evaluated against expert annotators. The configuration using shared embeddings provided the best overall balance of metrics. While the Policy Gradient method achieved the highest $F_{0.5}$ scores, it highlighted a trade-off between catching a wider variety of errors and maintaining absolute accuracy.

Conclusion

This research establishes ConvS2S as a powerful architecture for Chinese Grammatical Error Correction. While the model excels at "fixed match" errors involving local word choices, global sentence structure errors remain a challenge. Future work will focus on incorporating structural and syntactic information, such as dependency trees, to better handle long-range word ordering errors and refining the reinforcement learning reward functions to maintain high precision.

Summary updated

BibTeX
@article{li2019chinese,
  author = {Si Li and Jianbo Zhao and Guirong Shi and Yuanpeng Tan and Huifang Xu and Guang Chen and Haibo Lan and Zhiqing Lin},
  title = {{Chinese Grammatical Error Correction Based on Convolutional Sequence to Sequence Model}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2917631},
}
Figure