Revisiting Pre-Trained Models for Chinese Natural Language Processing

Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, Guoping Hu

Abstract

Pre-trained language models like BERT have revolutionized Natural Language Processing (NLP), yet they suffer from a discrepancy between pre-training and fine-tuning due to the use of the [MASK] token. This paper introduces MacBERT (MLM as correction BERT), a refined pre-training strategy specifically optimized for Chinese NLP. By replacing the traditional masking task with a language correction task using synonyms, and incorporating whole-word and N-gram masking, MacBERT achieves new state-of-the-art (SOTA) results across a wide array of Chinese benchmarks.

Introduction

While BERT and its successors (RoBERTa, ALBERT) have established a new paradigm in NLP, two significant challenges remain, particularly for non-English languages like Chinese. First, the Pre-train-Fine-tune Discrepancy: during pre-training, BERT masks 15% of tokens with a special [MASK] symbol that never appears during downstream fine-tuning. This creates a mismatch between the model's training environment and its actual application. Second, early models often relied on character-level masking, which fails to capture the semantic nuances of multi-character Chinese words.

Proposed Method: MacBERT

To address these issues, the authors propose MacBERT, which reimagines Masked Language Modeling (MLM) as a "correction" task.

MLM as Correction (Mac)

Instead of replacing a selected token with the [MASK] symbol, MacBERT replaces it with a similar word (synonym) identified via word2vec-based similarity. This forces the model to identify and "correct" the word based on context, making the pre-training task more natural and closer to downstream fine-tuning where all words are real. Of the 15% selected tokens, 80% are replaced with synonyms, 10% with random words, and 10% are kept as-is.

Comparison of various masking strategies in Chinese and English, highlighting the progression from character-level to Mac masking.

Chinese-Specific Optimizations

MacBERT incorporates several strategies to better handle Chinese linguistics:

  • Whole Word Masking (WWM) & N-gram Masking: The model masks entire words rather than individual characters and uses N-gram masking (sequences of 1 to 4 words) to capture broader context.
  • Sentence Order Prediction (SOP): Replacing the standard "Next Sentence Prediction" (NSP) task, MacBERT uses SOP (as seen in ALBERT), where the model determines if two segments are in their original order or swapped. This task is more robust as it requires understanding coherence rather than just topic matching.

Experimental Setup

The authors pre-trained MacBERT on a massive scale using Chinese Wikipedia (0.4 billion words) and an extended in-house collection ("ext") of encyclopedia, news, and QA data totaling 5.4 billion words. The model was evaluated on eight major benchmarks, including Machine Reading Comprehension (CMRC 2018, DRCD, CJRC), Single Sentence Classification (ChnSentiCorp, THUCNews), and Sentence Pair Classification (XNLI, LCQMC, BQ Corpus).

Key Results and Insights

MacBERT consistently outperformed original BERT and RoBERTa-wwm across almost all tasks. MacBERT-large achieved a 90.7 F1 score on the CMRC 2018 test set and a 95.6 F1 score on DRCD.

Ablation study showing MacBERT's performance (Exact Match) compared to other masking strategies across training steps.

Ablation studies (visualized above) confirmed that the "correction" approach provides a significant advantage over standard masking. Specifically, N-gram masking proved particularly effective for text classification, while similar word replacement provided the largest boost to reading comprehension tasks. The results suggest that the quality of the pre-training task—bridging the gap to real-world usage—is as critical as the quantity of parameters.

Conclusion

MacBERT demonstrates that rethinking the fundamental masking strategy can lead to significant gains in model performance and generalization. By treating MLM as a correction task and optimizing for Chinese word structures, MacBERT establishes a new baseline for Chinese NLP. Future work may involve investigating dynamic masking ratios and more efficient distillation methods for production environments.

Summary updated

BibTeX
@inproceedings{cui2020revisiting,
  author = {Yiming Cui and Wanxiang Che and Ting Liu and Bing Qin and Shijin Wang and Guoping Hu},
  title = {{Revisiting Pre-Trained Models for Chinese Natural Language Processing}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=8JRFShAsPZ},
}
Figure