Abstract
Bidirectional Encoder Representations from Transformers (BERT) has shown marvelous improvements across various NLP tasks, and consecutive variants have been proposed to further improve the performance of the pre-trained language models. In this paper, we target on revisiting Chinese pre-trained language models to examine their effectiveness in a non-English language and release the Chinese pre-trained language model series to the community. We also propose a simple but effective model called MacBERT, which improves upon RoBERTa in several ways, especially the masking strategy that adopts MLM as correction (Mac). We carried out extensive experiments on eight Chinese NLP tasks to revisit the existing pre-trained language models as well as the proposed MacBERT. Experimental results show that MacBERT could achieve state-of-the-art performances on many NLP tasks, and we also ablate details with several findings that may help future research.
Introduction
The advent of Transformer-based models like BERT has revolutionized the field of Natural Language Processing. However, the majority of research and optimization has been centered on the English language. Adapting these powerful architectures to Chinese presents unique challenges, such as the lack of explicit word boundaries and the structural nuances of the language.
A critical issue identified in standard BERT training is the pre-train/fine-tune discrepancy. During pre-training, the model learns to predict a special [MASK] token. However, this token is absent during fine-tuning on downstream tasks, creating a mismatch that can limit the model's generalization capabilities. Furthermore, character-level masking often fails to capture the semantic richness of multi-character Chinese words.
Methodology: The MacBERT Architecture
To address these limitations, we propose MacBERT (MLM as Correction BERT). MacBERT builds upon the RoBERTa architecture but introduces three fundamental changes to the pre-training objective.
MLM as Correction (Mac)
The core innovation of MacBERT is the "MLM as Correction" strategy. Instead of replacing a selected token with the artificial [MASK] token, we replace it with a synonym. For the 15% of tokens selected for masking, 80% are replaced with synonyms (identified via word2vec similarity), 10% with random words, and 10% remain unchanged. This shifts the model's objective from "predicting a missing token" to "detecting and correcting an incorrect token," a task much closer to natural language understanding and downstream applications.
Whole Word Masking (WWM) and N-gram Masking
To better handle Chinese morphology, we employ Chinese Word Segmentation (CWS) to perform Whole Word Masking. Additionally, we incorporate N-gram masking, where sequences of up to four words are masked together. This prevents the model from relying on local character patterns and forces it to learn higher-level semantic concepts.
Sentence Order Prediction (SOP)
We replace the traditional Next Sentence Prediction (NSP) task with Sentence Order Prediction (SOP). While NSP has been criticized for being too simple, SOP requires the model to distinguish between the original order of two consecutive sentences and their swapped version. This encourages the model to develop a deeper understanding of document coherence and logical flow.
Experimental Setup
MacBERT was pre-trained on a massive corpus comprising Chinese Wikipedia (0.4 billion words) and an extended dataset of encyclopedia, news, and QA web data totaling 5.4 billion words.
We evaluated the model across eight diverse Chinese benchmarks:
- Machine Reading Comprehension (MRC): CMRC 2018, DRCD, CJRC.
- Single Sentence Classification (SSC): ChnSentiCorp, THUCNews.
- Sentence Pair Classification (SPC): XNLI, LCQMC, BQ Corpus.
Results and Discussion
MacBERT achieved state-of-the-art (SOTA) results on nearly all tested benchmarks. For instance, on the CMRC 2018 reading comprehension task, MacBERT-large reached an F1 score of 90.7, a significant improvement over the original BERT-base score of 87.0.
Ablation Study: The Power of Correction
Ablation studies confirmed that the "Mac" strategy is the primary driver of performance gains. As shown in the figure below, replacing the [MASK] token with synonyms or random words consistently outperforms the standard masking approach as training progresses.
Our findings suggest that the [MASK] token acts as a "crutch" that hinders the model's ability to learn robust contextual representations. By forcing the model to perform "grammar correction," it develops features that translate more effectively to real-world tasks.
Conclusion and Future Work
MacBERT demonstrates that eliminating the pre-train/fine-tune discrepancy through synonym-based correction significantly enhances performance in Chinese NLP. While highly effective, future work could explore more dynamic masking ratios beyond the heuristic 15%. Additionally, while MacBERT excels in general domains, there remains significant potential for improvement in specialized fields like legal or medical text through domain-specific pre-training.
Resources available: https://github.com/ymcui/MacBERT
Summary updated