Abstract
Chinese Natural Language Processing (NLP) faces a unique challenge due to the lack of explicit word boundaries, leading to a persistent debate between character-based and word-based tokenization. This paper introduces MVP-BERT, which utilizes a Multi-Vocab pre-training strategy to bridge this gap. By incorporating Segmentation Guided Tokenization (SGT) and an Auxiliary Loss Multi-Vocab Pre-training (AL-MVP) architecture, the model captures both fine-grained character details and coarse-grained word semantics. MVP-BERT achieves superior performance across various benchmarks, offers a 2.15x inference speedup over multi-grained competitors like AMBERT, and demonstrates enhanced robustness against adversarial attacks.
Introduction
In English NLP, sub-word tokenization is standard. However, Chinese text lacks spaces, forcing a choice between character-level and word-level representations. Character-based models are robust against Out-of-Vocabulary (OOV) issues but often fail to capture complex semantic meanings. Conversely, word-based models leverage Chinese Word Segmentation (CWS) to capture richer semantics but are vulnerable to segmentation errors and rare word issues. Previous attempts to combine these granularities, such as AMBERT, often resulted in doubled computational costs due to dual-encoder architectures. MVP-BERT addresses these limitations by integrating multi-granularity information into a single, efficient encoder.
Methodology
Segmentation Guided Tokenization (SGT)
To find a "sweet spot" between characters and words, the authors propose SGT. This method first segments text using a CWS tool and then applies Byte Pair Encoding (BPE) to the resulting segments. This creates a vocabulary that preserves common multi-character phrases while decomposing rare words into smaller, manageable units.
Model Architectures
The paper explores two primary architectures for multi-vocab integration:
- Hierarchical MVP (Hi-MVP): This model uses a CNN and max-pooling layer to aggregate character embeddings into word-level representations before processing them through Transformer layers.
- Auxiliary Loss MVP (AL-MVP): This is the flagship contribution. It employs a single Transformer encoder trained with an auxiliary Masked Language Modeling (MLM) loss. During pre-training, the model simultaneously predicts individual characters and the full word for a masked span. Crucially, the multi-vocab setup is only required during training; at inference, it functions as a standard, highly efficient BERT model.
Experimental Setup
Models were pre-trained on two corpora: C-1 (Chinese Wikipedia) and C-2 (a 25-million document corpus of Wikipedia and news). Evaluation spanned six benchmark tasks, including Classification (XNLI, LCQMC), Named Entity Recognition (MSRA, CCKS), and Machine Reading Comprehension (CMRC2018, ChID).
Results and Discussion
Performance and Efficiency
AL-MVP consistently outperformed baseline Google Chinese BERT and AMBERT. Notably, on the ChID idiom task, it achieved 86.76% accuracy, significantly higher than Google BERT's 82.04%. Because SGT collapses multiple characters into single tokens, the resulting sequences are shorter, leading to a 1.25x inference speedup over character-based models. Compared to the dual-encoder AMBERT, AL-MVP provides a 2.15x speedup while requiring only half the training resources.
Adversarial Robustness
A key finding is that multi-vocab training enhances model stability. Under the TextFooler adversarial framework, AL-MVP maintained higher accuracy and proved more difficult to "fool" than single-vocab baselines, suggesting it learns more generalized linguistic representations.
Conclusion
MVP-BERT demonstrates that multi-granularity information can be effectively integrated into Chinese language models without sacrificing efficiency. Future research will explore more sophisticated aggregation methods and ways to reduce dependency on initial CWS tools to further mitigate error propagation.
Summary updated