Shuo Wen Jie Zi: Rethinking Dictionaries and Glyphs for Chinese Language Pre-training

Yuxuan Wang, Jianghui Wang, Dongyan Zhao, Zilong Zheng

TL;DR

CDBERT is a new pre-training paradigm that injects dictionary definitions and character structural information (glyphs) into Chinese language models. By simulating how humans use dictionaries to disambiguate polysemy and understand rare characters, it achieves significant performance gains across modern and ancient Chinese tasks, particularly in challenging few-shot scenarios.

What Problem Does This Paper Address?

While large-scale pre-trained language models (PLMs) like BERT and GPT have transformed NLP, Chinese presents unique challenges that standard architectures often overlook:

  1. The Long Tail of Characters: Unlike the 26 letters of the English alphabet, Chinese has thousands of characters. New or rare characters appear frequently, requiring models to adapt quickly.
  2. Pervasive Polysemy: A single Chinese character often has multiple, distinct meanings depending on the context.
  3. Logographic Structure: Chinese characters aren't just arbitrary symbols; they are composed of radicals and strokes that carry inherent semantic meaning.

Existing models often treat characters as atomic tokens or flat images, missing the rich semantic "DNA" found in dictionaries and character structures. The authors address this by looking back at the Shuō Wén Jiě Zì—the foundational 2nd-century Chinese dictionary—to inspire a model that truly "understands" the language's building blocks.

Methods

The authors introduce CDBERT, which consists of two core modules named after the classical dictionary:

  • Shuowen (Dictionary Module): This module treats a dictionary as a knowledge base. It uses three pre-training tasks to help the model learn word meanings:
  • Masked Entry Modeling (MEM): The model must predict a word given its definition.
  • Contrastive Learning (CL4SA): It learns to pull synonyms closer and push antonyms further apart in the embedding space.
  • Example Learning (EL): This is the "aha" module for polysemy. Given an example sentence, the model uses an attention mechanism to select the most appropriate definition from a list of candidates.
  • Jiezi (Glyph Module): This enhances character representations using their visual and structural properties.
  • It uses Radical-based embeddings to decompose characters into their semantic components (e.g., the character "明" is composed of "日" [sun] and "月" [moon]).
  • It also leverages CLIP-enhanced glyph representations, using vision-language pre-training to align images of characters (in various fonts and real-world settings) with their semantic definitions.
Figure 1: Illustration of CDBERT. The expression in green refers to the selected definition of current charac- ter.

As shown in Figure 3, these dictionary-augmented features are fused with standard PLM outputs to provide a richer context for downstream tasks.

Integration of CDBERT into downstream tasks.

Datasets

The model was evaluated on:

  • CLUE: The standard benchmark for modern Chinese understanding.
  • CCLUE: A benchmark for ancient (classical) Chinese.
  • PolyMRC (New): The authors introduced this new task specifically for polysemy discrimination. It requires the model to choose the correct definition for a word used in a specific example sentence, sourced from BaiduHanyu.
Table 1: Statistics of PolyMRC Dataset

Key Insights and Ideas

  • Dictionaries as Semantic Anchors: Simply adding more text corpus isn't enough for deep understanding. Dictionaries provide a structured "ground truth" that helps models navigate the ambiguity of polysemy.
  • Structure Over Pixels: The ablation studies revealed that radical-based embeddings are often more effective than raw pixel-based glyph features. This suggests that the symbolic structure of a character is more semantically informative than its raw visual appearance when training data is limited.
  • Few-Shot Prowess: CDBERT shines brightest when data is scarce. By "reading" the dictionary, the model gains a prior understanding that allows it to perform well on ancient Chinese tasks with only a handful of examples.

Key Results

CDBERT consistently outperformed baselines (BERT, RoBERTa, MacBERT) across all benchmarks:

  • Modern Chinese (CLUE): Achieved an average improvement of 1.8% on single-sentence classification tasks.
  • Ancient Chinese (CCLUE): Showed a significant boost in Machine Reading Comprehension (MRC), improving average accuracy from 42.93 to 44.72.
  • Few-Shot Learning: In 10-shot settings, CDBERT+BERT improved accuracy by 18.3% on PolyMRC and 18.45% on CCLUE-MRC compared to standard BERT.
Table 4: Performance of CDBERT on CCLUE.

Limitations and Future Work

The authors acknowledge several areas for improvement:

  • Dictionary Quality: The current dictionary data, sourced from the internet, contains some noise and lacks total coverage of all Chinese words.
  • Architecture Constraints: The method is currently designed for BERT-like encoder models. Adapting this dictionary-infusion technique to modern Large Language Models (LLMs) like GPT-4 remains a future challenge.
  • Bilingual Expansion: The researchers plan to explore bilingual dictionaries and more fine-grained structural information to further refine the model's understanding and generation capabilities.

Summary updated

Abstract

We introduce CDBERT, a new learning paradigm that enhances the semantics understanding ability of the Chinese PLMs with dictionary knowledge and structure of Chinese characters. We name the two core modules of CDBERT as Shuowen and Jiezi, where Shuowen refers to the process of retrieving the most appropriate meaning from Chinese dictionaries and Jiezi refers to the process of enhancing characters' glyph representations with structure understanding. To facilitate dictionary understanding, we propose three pre-training tasks, i.e., Masked Entry Modeling, Contrastive Learning for Synonym and Antonym, and Example Learning. We evaluate our method on both modern Chinese understanding benchmark CLUE and ancient Chinese benchmark CCLUE. Moreover, we propose a new polysemy discrimination task PolyMRC based on the collected dictionary of ancient Chinese. Our paradigm demonstrates consistent improvements on previous Chinese PLMs across all tasks. Moreover, our approach yields significant boosting on few-shot setting of ancient Chinese understanding.
BibTeX
@inproceedings{wang2023shuo,
  author = {Yuxuan Wang and Jianghui Wang and Dongyan Zhao and Zilong Zheng},
  title = {{Shuo Wen Jie Zi: Rethinking Dictionaries and Glyphs for Chinese Language Pre-training}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.70},
}
Figure