Rethinking Dictionaries and Glyphs for Chinese Language Pre-training

Yuxuan Wang, Jianghui Wang, Dongyan Zhao, Zilong Zheng

Introduction

Large-scale pre-trained language models (PLMs) like BERT and GPT have revolutionized natural language processing. However, applying these models to the Chinese language presents unique challenges. Unlike alphabetic languages, Chinese is logographic and highly polysemous. Current models often struggle with the "long tail" of rare characters, the rampant ambiguity of polysemous words, and the internal structural information of characters—the radicals—which often carry essential semantic meaning.

To address these issues, we introduce CDBERT, a novel pre-training paradigm named after the classic Chinese dictionary Shuō Wén Jiě Zì. CDBERT integrates structured dictionary knowledge and character glyph information, mimicking how humans use dictionaries to resolve ambiguity and understand complex characters.

The CDBERT Framework

CDBERT is composed of two primary modules designed to capture the semantic and structural nuances of the Chinese language.

Shuowen: The Dictionary Module

The Shuowen module treats dictionaries as high-quality, human-curated knowledge bases. It utilizes three specialized tasks to ground the model's embeddings in formal definitions:

  • Masked Entry Modeling (MEM): The model is tasked with predicting a character "entry" given its dictionary definition, establishing a direct link between semantic descriptions and logographs.
  • Contrastive Learning for Synonym and Antonym (CL4SA): This task forces the model to cluster synonyms together and push antonyms apart in the vector space, refining its understanding of word relationships.
  • Example Learning (EL): Using an attention mechanism, the model learns to select the correct definition for a character based on its specific usage in a sentence, directly addressing the problem of polysemy.

Jiezi: The Glyph Module

The Jiezi module focuses on the visual and structural components of Chinese characters. By leveraging Contrastive Language-Image Pre-training (CLIP), the model aligns character images—including various fonts and real-world captures—with their textual definitions.

The Jiezi module architecture for aligning visual glyphs with semantic definitions.

As illustrated in the figure above, the module explicitly incorporates radical-based embeddings. By decomposing characters into their constituent radicals (e.g., '明' into '日' and '月'), the model gains a native-like ability to infer the meaning of unfamiliar or rare characters based on their structural "DNA."

Evaluation and Datasets

CDBERT was evaluated on several key benchmarks:

  • CLUE: The standard benchmark for modern Chinese Natural Language Understanding (NLU).
  • CCLUE: A benchmark focused on ancient (classical) Chinese.
  • PolyMRC: A new dataset introduced by the authors to specifically test a model's ability to distinguish between multiple meanings of the same word in ancient Chinese, containing over 46,000 training sentences.

Key Results

The integration of dictionary and glyph knowledge yielded significant performance gains across the board. On the CLUE benchmark, CDBERT consistently outperformed standard BERT and RoBERTa baselines.

Performance comparison on CLUE benchmarks across different base models.

As shown in the results table, the RoBERTa_ext-large + CDBERT model achieved the highest overall score of 74.95. The model showed an average improvement of 1.8% on single-sentence tasks. The gains were even more pronounced in ancient Chinese, where accuracy across four models improved from 42.93 to 44.72.

Perhaps most impressively, CDBERT demonstrated robust few-shot performance. In a 10-shot setting on the PolyMRC task, BERT+CDBERT improved accuracy from 30.98% to 36.65%—a relative jump of over 18%. This suggests that dictionary knowledge acts as a powerful "expert prior," allowing the model to navigate ambiguity even when labeled data is scarce.

Conclusion

CDBERT represents a significant step forward in Chinese language pre-training by moving beyond raw text and incorporating the structured knowledge of dictionaries and the visual logic of glyphs. While challenges remain in scaling this approach to massive generative models and handling dictionary noise, the results clearly demonstrate the value of "rethinking" how models learn logographic languages.

Summary updated

Abstract

We introduce CDBert, a new learning paradigm that enhances the semantics understanding ability of the Chinese PLMs with dictionary knowledge and structure of Chinese characters. We name the two core modules of CDBert as Shuowen and Jiezi, where Shuowen refers to the process of retrieving the most appropriate meaning from Chinese dictionaries and Jiezi refers to the process of enhancing characters’ glyph representations with structure understanding. To facilitate dictionary understanding, we propose three pre-training tasks, i.e.„ Masked Entry Modeling, Contrastive Learning for Synonym and Antonym, and Example Learning. We evaluate our method on both modern Chinese understanding benchmark CLUE and ancient Chinese benchmark CCLUE. Moreover, we propose a new polysemy discrimination task PolyMRC based on the collected dictionary of ancient Chinese. Our paradigm demonstrates consistent improvements on previous Chinese PLMs across all tasks. Moreover, our approach yields significant boosting on few-shot setting of ancient Chinese understanding.
BibTeX
@inproceedings{wang2023rethinking,
  author = {Yuxuan Wang and Jianghui Wang and Dongyan Zhao and Zilong Zheng},
  title = {{Rethinking Dictionaries and Glyphs for Chinese Language Pre-training}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.70},
}
Figure