MVP-BERT: Multi-Vocab Pre-training for Chinese BERT

Wei Zhu

Abstract

Chinese Natural Language Processing (NLP) faces a unique challenge due to the lack of explicit word boundaries, leading to a persistent debate between character-based and word-based tokenization. This paper introduces MVP-BERT, which utilizes a Multi-Vocab pre-training strategy to bridge this gap. By incorporating Segmentation Guided Tokenization (SGT) and an Auxiliary Loss Multi-Vocab Pre-training (AL-MVP) architecture, the model captures both fine-grained character details and coarse-grained word semantics. MVP-BERT achieves superior performance across various benchmarks, offers a 2.15x inference speedup over multi-grained competitors like AMBERT, and demonstrates enhanced robustness against adversarial attacks.

Introduction

In English NLP, sub-word tokenization is standard. However, Chinese text lacks spaces, forcing a choice between character-level and word-level representations. Character-based models are robust against Out-of-Vocabulary (OOV) issues but often fail to capture complex semantic meanings. Conversely, word-based models leverage Chinese Word Segmentation (CWS) to capture richer semantics but are vulnerable to segmentation errors and rare word issues. Previous attempts to combine these granularities, such as AMBERT, often resulted in doubled computational costs due to dual-encoder architectures. MVP-BERT addresses these limitations by integrating multi-granularity information into a single, efficient encoder.

Methodology

Segmentation Guided Tokenization (SGT)

To find a "sweet spot" between characters and words, the authors propose SGT. This method first segments text using a CWS tool and then applies Byte Pair Encoding (BPE) to the resulting segments. This creates a vocabulary that preserves common multi-character phrases while decomposing rare words into smaller, manageable units.

Comparison of tokenization strategies including SEG, SGT, Hi-MVP, and CHAR.

Model Architectures

The paper explores two primary architectures for multi-vocab integration:

  1. Hierarchical MVP (Hi-MVP): This model uses a CNN and max-pooling layer to aggregate character embeddings into word-level representations before processing them through Transformer layers.
  2. Auxiliary Loss MVP (AL-MVP): This is the flagship contribution. It employs a single Transformer encoder trained with an auxiliary Masked Language Modeling (MLM) loss. During pre-training, the model simultaneously predicts individual characters and the full word for a masked span. Crucially, the multi-vocab setup is only required during training; at inference, it functions as a standard, highly efficient BERT model.
Architectural comparison between (a) Hi-MVP, (b) AL-MVP, and (c) the baseline AMBERT.

Experimental Setup

Models were pre-trained on two corpora: C-1 (Chinese Wikipedia) and C-2 (a 25-million document corpus of Wikipedia and news). Evaluation spanned six benchmark tasks, including Classification (XNLI, LCQMC), Named Entity Recognition (MSRA, CCKS), and Machine Reading Comprehension (CMRC2018, ChID).

Results and Discussion

Performance and Efficiency

AL-MVP consistently outperformed baseline Google Chinese BERT and AMBERT. Notably, on the ChID idiom task, it achieved 86.76% accuracy, significantly higher than Google BERT's 82.04%. Because SGT collapses multiple characters into single tokens, the resulting sequences are shorter, leading to a 1.25x inference speedup over character-based models. Compared to the dual-encoder AMBERT, AL-MVP provides a 2.15x speedup while requiring only half the training resources.

Adversarial Robustness

A key finding is that multi-vocab training enhances model stability. Under the TextFooler adversarial framework, AL-MVP maintained higher accuracy and proved more difficult to "fool" than single-vocab baselines, suggesting it learns more generalized linguistic representations.

Conclusion

MVP-BERT demonstrates that multi-granularity information can be effectively integrated into Chinese language models without sacrificing efficiency. Future research will explore more sophisticated aggregation methods and ways to reduce dependency on initial CWS tools to further mitigate error propagation.

Summary updated

Abstract

Despite the development of pre-trained language models (PLMs) significantly raise the performances of various Chinese natural language processing (NLP) tasks, the vocabulary (vocab) for these Chinese PLMs remains to be the one provided by Google Chinese BERT (CITATION), which is based on Chinese characters (chars). Second, the masked language model pre-training is based on a single vocab, limiting its downstream task performances. In this work, we first experimentally demonstrate that building a vocab via Chinese word segmentation (CWS) guided sub-word tokenization (SGT) can improve the performances of Chinese PLMs. Then we propose two versions of multi-vocab pre-training (MVP), Hi-MVP and AL-MVP, to improve the models’ expressiveness. Experiments show that: (a) MVP training strategies improve PLMs’ downstream performances, especially it can improve the PLM’s performances on span-level tasks; (b) our AL-MVP outperforms the recent AMBERT (CITATION) after large-scale pre-training, and it is more robust against adversarial attacks.
BibTeX
@inproceedings{zhu2021mvp,
  author = {Wei Zhu},
  title = {{MVP-BERT: Multi-Vocab Pre-training for Chinese BERT}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-srw.27},
}
Figure