Integrating BERT and Score-based Feature Gates for Chinese Grammatical Error Diagnosis

Yongchang Cao, Liang He, Robert Ridley, Xinyu Dai2020 · DOI

Abstract

This paper describes the BERT with Score-feature Gates Error Diagnoser (BSGED) model proposed for the Chinese Grammatical Error Diagnosis (CGED) task at NLPTEA 2020. BSGED integrates BERT, Bidirectional Long Short-Term Memory (BiLSTM), and Conditional Random Fields (CRF). A key innovation is a novel gating mechanism for continuous feature items, such as Pointwise Mutual Information (PMI) and Part-of-Speech (POS) scores, which preserves partial-order relationships often lost during discretization. Evaluation on the official test set shows BSGED achieving the highest F1 score at the detection level and ranking among the top three at the identification level.

Introduction

Chinese Grammatical Error Diagnosis (CGED) involves identifying four error types: redundant words (R), missing words (M), word selection (S), and word ordering (W). While deep learning models like BiLSTM-CRF are standard, they often struggle to integrate continuous linguistic features effectively.

Traditionally, continuous scores (e.g., PMI) are discretized into bins and embedded. The authors identify two major drawbacks to this approach:

  1. Information Loss: Discretization destroys the "partial-order" relationship (e.g., the model loses the knowledge that 0.8 is closer to 0.9 than to 0.1).
  2. Parameter Inefficiency: High-granularity bins lead to massive embedding matrices, complicating training.

The BSGED Architecture

The BSGED model is designed to leverage both deep contextual embeddings and fine-grained linguistic scores without the pitfalls of discretization.

The overall architecture of the BSGED model, showing the flow from input tokens to features, BiLSTM, and the final CRF layer.

The architecture consists of several key layers:

  • BERT Encoder: Utilizes BERT-wwm-ext (Whole Word Masking) for robust semantic feature extraction.
  • BiLSTM Layer: Processes BERT outputs to capture long-term sequence dependencies specific to sequence labeling.
  • Gating Mechanism: Instead of binning, continuous scores are treated as scalar gates. A small position-based embedding matrix is multiplied by the raw score, maintaining the mathematical relationship between values.
  • CRF Layer: Predicts the final sequence of error tags.

Experimental Results

The model was evaluated on the CGED 2020 test set after training on a consolidated dataset from 2015-2020 (approximately 44,000 samples).

Performance comparison of BSGED (Run #1) against the best-performing teams at various diagnosis levels.

Key findings include:

  • State-of-the-Art Performance: BSGED achieved the highest F1 score (0.9122) at the Detection level and ranked in the top 3 at the Identification level (F1: 0.6736).
  • Ablation Benefits: The gating mechanism outperformed traditional discretization by approximately 3.7% in F1 at the identification level. Additionally, the BiLSTM layer provided a 1.5% to 3% F1 boost over BERT-only configurations.
  • Ensemble Strategy: Using a voting mechanism across 16 model initializations and rule-based post-processing further refined the results.

Conclusion

BSGED demonstrates that preserving the continuity of linguistic features through a gating mechanism significantly improves grammatical error diagnosis. While highly effective at detection and identification, future work will focus on error correction using Masked Language Models and the integration of explicit grammatical rules to handle complex semantic edge cases.

Reference: Cao, Y., He, L., Ridley, R., & Dai, X. (2020). Integrating BERT and Score-based Feature Gates for Chinese Grammatical Error Diagnosis. Proceedings of the 6th Workshop on Natural Language Processing Techniques for Educational Applications, 49–56.

Summary updated

Abstract

This paper describes our proposed model for the Chinese Grammatical Error Diagnosis (CGED) task in NLPTEA2020. The goal of CGED is to use natural language processing techniques to automatically diagnose Chinese grammatical errors in sentences. To this end, we design and implement a CGED model named BERT with Score-feature Gates Error Diagnoser (BSGED), which is based on the BERT model, Bidirectional Long Short-Term Memory (BiLSTM) and conditional random field (CRF). In order to address the problem of losing partial-order relationships when embedding continuous feature items as with previous works, we propose a gating mechanism for integrating continuous feature items, which effectively retains the partial-order relationships between feature items. We perform LSTM processing on the encoding result of the BERT model, and further extract the sequence features. In the final test-set evaluation, we obtained the highest F1 score at the detection level and are among the top 3 F1 scores at the identification level.
BibTeX
@misc{cao2020integrating,
  author = {Yongchang Cao and Liang He and Robert Ridley and Xinyu Dai},
  title = {{Integrating BERT and Score-based Feature Gates for Chinese Grammatical Error Diagnosis}},
  year = {2020},
  doi = {10.18653/v1/2020.nlptea-1.7},
}
Figure