TL;DR
CDBERT is a new pre-training paradigm that injects dictionary definitions and character structural information (glyphs) into Chinese language models. By simulating how humans use dictionaries to disambiguate polysemy and understand rare characters, it achieves significant performance gains across modern and ancient Chinese tasks, particularly in challenging few-shot scenarios.
What Problem Does This Paper Address?
While large-scale pre-trained language models (PLMs) like BERT and GPT have transformed NLP, Chinese presents unique challenges that standard architectures often overlook:
- The Long Tail of Characters: Unlike the 26 letters of the English alphabet, Chinese has thousands of characters. New or rare characters appear frequently, requiring models to adapt quickly.
- Pervasive Polysemy: A single Chinese character often has multiple, distinct meanings depending on the context.
- Logographic Structure: Chinese characters aren't just arbitrary symbols; they are composed of radicals and strokes that carry inherent semantic meaning.
Existing models often treat characters as atomic tokens or flat images, missing the rich semantic "DNA" found in dictionaries and character structures. The authors address this by looking back at the Shuō Wén Jiě Zì—the foundational 2nd-century Chinese dictionary—to inspire a model that truly "understands" the language's building blocks.
Methods
The authors introduce CDBERT, which consists of two core modules named after the classical dictionary:
- Shuowen (Dictionary Module): This module treats a dictionary as a knowledge base. It uses three pre-training tasks to help the model learn word meanings:
- Masked Entry Modeling (MEM): The model must predict a word given its definition.
- Contrastive Learning (CL4SA): It learns to pull synonyms closer and push antonyms further apart in the embedding space.
- Example Learning (EL): This is the "aha" module for polysemy. Given an example sentence, the model uses an attention mechanism to select the most appropriate definition from a list of candidates.
- Jiezi (Glyph Module): This enhances character representations using their visual and structural properties.
- It uses Radical-based embeddings to decompose characters into their semantic components (e.g., the character "明" is composed of "日" [sun] and "月" [moon]).
- It also leverages CLIP-enhanced glyph representations, using vision-language pre-training to align images of characters (in various fonts and real-world settings) with their semantic definitions.
As shown in Figure 3, these dictionary-augmented features are fused with standard PLM outputs to provide a richer context for downstream tasks.
Datasets
The model was evaluated on:
- CLUE: The standard benchmark for modern Chinese understanding.
- CCLUE: A benchmark for ancient (classical) Chinese.
- PolyMRC (New): The authors introduced this new task specifically for polysemy discrimination. It requires the model to choose the correct definition for a word used in a specific example sentence, sourced from BaiduHanyu.
Key Insights and Ideas
- Dictionaries as Semantic Anchors: Simply adding more text corpus isn't enough for deep understanding. Dictionaries provide a structured "ground truth" that helps models navigate the ambiguity of polysemy.
- Structure Over Pixels: The ablation studies revealed that radical-based embeddings are often more effective than raw pixel-based glyph features. This suggests that the symbolic structure of a character is more semantically informative than its raw visual appearance when training data is limited.
- Few-Shot Prowess: CDBERT shines brightest when data is scarce. By "reading" the dictionary, the model gains a prior understanding that allows it to perform well on ancient Chinese tasks with only a handful of examples.
Key Results
CDBERT consistently outperformed baselines (BERT, RoBERTa, MacBERT) across all benchmarks:
- Modern Chinese (CLUE): Achieved an average improvement of 1.8% on single-sentence classification tasks.
- Ancient Chinese (CCLUE): Showed a significant boost in Machine Reading Comprehension (MRC), improving average accuracy from 42.93 to 44.72.
- Few-Shot Learning: In 10-shot settings, CDBERT+BERT improved accuracy by 18.3% on PolyMRC and 18.45% on CCLUE-MRC compared to standard BERT.
Limitations and Future Work
The authors acknowledge several areas for improvement:
- Dictionary Quality: The current dictionary data, sourced from the internet, contains some noise and lacks total coverage of all Chinese words.
- Architecture Constraints: The method is currently designed for BERT-like encoder models. Adapting this dictionary-infusion technique to modern Large Language Models (LLMs) like GPT-4 remains a future challenge.
- Bilingual Expansion: The researchers plan to explore bilingual dictionaries and more fine-grained structural information to further refine the model's understanding and generation capabilities.
Summary updated