Abstract
This paper describes the BERT with Score-feature Gates Error Diagnoser (BSGED) model proposed for the Chinese Grammatical Error Diagnosis (CGED) task at NLPTEA 2020. BSGED integrates BERT, Bidirectional Long Short-Term Memory (BiLSTM), and Conditional Random Fields (CRF). A key innovation is a novel gating mechanism for continuous feature items, such as Pointwise Mutual Information (PMI) and Part-of-Speech (POS) scores, which preserves partial-order relationships often lost during discretization. Evaluation on the official test set shows BSGED achieving the highest F1 score at the detection level and ranking among the top three at the identification level.
Introduction
Chinese Grammatical Error Diagnosis (CGED) involves identifying four error types: redundant words (R), missing words (M), word selection (S), and word ordering (W). While deep learning models like BiLSTM-CRF are standard, they often struggle to integrate continuous linguistic features effectively.
Traditionally, continuous scores (e.g., PMI) are discretized into bins and embedded. The authors identify two major drawbacks to this approach:
- Information Loss: Discretization destroys the "partial-order" relationship (e.g., the model loses the knowledge that 0.8 is closer to 0.9 than to 0.1).
- Parameter Inefficiency: High-granularity bins lead to massive embedding matrices, complicating training.
The BSGED Architecture
The BSGED model is designed to leverage both deep contextual embeddings and fine-grained linguistic scores without the pitfalls of discretization.
The architecture consists of several key layers:
- BERT Encoder: Utilizes BERT-wwm-ext (Whole Word Masking) for robust semantic feature extraction.
- BiLSTM Layer: Processes BERT outputs to capture long-term sequence dependencies specific to sequence labeling.
- Gating Mechanism: Instead of binning, continuous scores are treated as scalar gates. A small position-based embedding matrix is multiplied by the raw score, maintaining the mathematical relationship between values.
- CRF Layer: Predicts the final sequence of error tags.
Experimental Results
The model was evaluated on the CGED 2020 test set after training on a consolidated dataset from 2015-2020 (approximately 44,000 samples).
Key findings include:
- State-of-the-Art Performance: BSGED achieved the highest F1 score (0.9122) at the Detection level and ranked in the top 3 at the Identification level (F1: 0.6736).
- Ablation Benefits: The gating mechanism outperformed traditional discretization by approximately 3.7% in F1 at the identification level. Additionally, the BiLSTM layer provided a 1.5% to 3% F1 boost over BERT-only configurations.
- Ensemble Strategy: Using a voting mechanism across 16 model initializations and rule-based post-processing further refined the results.
Conclusion
BSGED demonstrates that preserving the continuity of linguistic features through a gating mechanism significantly improves grammatical error diagnosis. While highly effective at detection and identification, future work will focus on error correction using Masked Language Models and the integration of explicit grammatical rules to handle complex semantic edge cases.
Reference: Cao, Y., He, L., Ridley, R., & Dai, X. (2020). Integrating BERT and Score-based Feature Gates for Chinese Grammatical Error Diagnosis. Proceedings of the 6th Workshop on Natural Language Processing Techniques for Educational Applications, 49–56.
Summary updated