Abstract
Pre-trained language models like BERT have revolutionized Natural Language Processing (NLP), yet they suffer from a discrepancy between pre-training and fine-tuning due to the use of the [MASK] token. This paper introduces MacBERT (MLM as correction BERT), a refined pre-training strategy specifically optimized for Chinese NLP. By replacing the traditional masking task with a language correction task using synonyms, and incorporating whole-word and N-gram masking, MacBERT achieves new state-of-the-art (SOTA) results across a wide array of Chinese benchmarks.
Introduction
While BERT and its successors (RoBERTa, ALBERT) have established a new paradigm in NLP, two significant challenges remain, particularly for non-English languages like Chinese. First, the Pre-train-Fine-tune Discrepancy: during pre-training, BERT masks 15% of tokens with a special [MASK] symbol that never appears during downstream fine-tuning. This creates a mismatch between the model's training environment and its actual application. Second, early models often relied on character-level masking, which fails to capture the semantic nuances of multi-character Chinese words.
Proposed Method: MacBERT
To address these issues, the authors propose MacBERT, which reimagines Masked Language Modeling (MLM) as a "correction" task.
MLM as Correction (Mac)
Instead of replacing a selected token with the [MASK] symbol, MacBERT replaces it with a similar word (synonym) identified via word2vec-based similarity. This forces the model to identify and "correct" the word based on context, making the pre-training task more natural and closer to downstream fine-tuning where all words are real. Of the 15% selected tokens, 80% are replaced with synonyms, 10% with random words, and 10% are kept as-is.
Chinese-Specific Optimizations
MacBERT incorporates several strategies to better handle Chinese linguistics:
- Whole Word Masking (WWM) & N-gram Masking: The model masks entire words rather than individual characters and uses N-gram masking (sequences of 1 to 4 words) to capture broader context.
- Sentence Order Prediction (SOP): Replacing the standard "Next Sentence Prediction" (NSP) task, MacBERT uses SOP (as seen in ALBERT), where the model determines if two segments are in their original order or swapped. This task is more robust as it requires understanding coherence rather than just topic matching.
Experimental Setup
The authors pre-trained MacBERT on a massive scale using Chinese Wikipedia (0.4 billion words) and an extended in-house collection ("ext") of encyclopedia, news, and QA data totaling 5.4 billion words. The model was evaluated on eight major benchmarks, including Machine Reading Comprehension (CMRC 2018, DRCD, CJRC), Single Sentence Classification (ChnSentiCorp, THUCNews), and Sentence Pair Classification (XNLI, LCQMC, BQ Corpus).
Key Results and Insights
MacBERT consistently outperformed original BERT and RoBERTa-wwm across almost all tasks. MacBERT-large achieved a 90.7 F1 score on the CMRC 2018 test set and a 95.6 F1 score on DRCD.
Ablation studies (visualized above) confirmed that the "correction" approach provides a significant advantage over standard masking. Specifically, N-gram masking proved particularly effective for text classification, while similar word replacement provided the largest boost to reading comprehension tasks. The results suggest that the quality of the pre-training task—bridging the gap to real-world usage—is as critical as the quantity of parameters.
Conclusion
MacBERT demonstrates that rethinking the fundamental masking strategy can lead to significant gains in model performance and generalization. By treating MLM as a correction task and optimizing for Chinese word structures, MacBERT establishes a new baseline for Chinese NLP. Future work may involve investigating dynamic masking ratios and more efficient distillation methods for production environments.
Summary updated