Abstract
Most of the Chinese pre-trained models adopt characters as basic units for downstream tasks. However, these models ignore the information carried by words and thus lead to the loss of some important semantics. In this paper, we propose a new method to exploit word structure and integrate lexical semantics into character representations of pre-trained models. Specifically, we project a word's embedding into its internal characters' embeddings according to the similarity weight. To strengthen the word boundary information, we mix the representations of the internal characters within a word. After that, we apply a word-to-character alignment attention mechanism to emphasize important characters by masking unimportant ones. Moreover, in order to reduce the error propagation caused by word segmentation, we present an ensemble approach to combine segmentation results given by different tokenizers. The experimental results show that our approach achieves superior performance over the basic pre-trained models BERT, BERT-wwm and ERNIE on different Chinese NLP tasks: sentiment classification, sentence pair matching, natural language inference and machine reading comprehension. We make further analysis to prove the effectiveness of each component of our model.
Introduction
Unlike English, where spaces clearly delineate words, Chinese is written as a continuous string of characters. Most popular pre-trained language models (PLMs) like BERT treat Chinese characters as the basic unit. However, a single Chinese character is often highly ambiguous; for instance, the character “足” can mean "foot," "attain," "satisfy," or "enough" depending on the context.
The true semantic meaning usually resides at the word level (combinations of characters). While researchers have attempted to integrate word information previously, they typically rely on "Splice" (adding words to the input sequence) or "Add" (adding word vectors to character vectors) methods. These approaches often increase computational overhead or introduce noise. This paper introduces the Hidden Representation Mix and Fusion (HRMF) framework to integrate word semantics intelligently without losing fine-grained character details.
Methodology
The HRMF framework operates during the fine-tuning stage and consists of four primary components designed to bridge the gap between character-level processing and word-level semantics.
Multiple Word Segmentation Aggregation
Word segmentation errors are a major source of error propagation in Chinese NLP. To mitigate this, the authors employ an ensemble approach using three different tokenizers: PKUSEG, LAC, and snownlp. By applying a majority-rule and granularity-rule voting system, the model determines more stable and accurate word boundaries.
The HRMF Architecture
Following segmentation, the framework integrates word semantics through three additional layers:
- Word Embedding Integration: The model calculates the cosine similarity between a character’s hidden state and its parent word's embedding. This "importance weight" determines how much semantic information is projected onto each character, avoiding the "smearing" of meaning across less relevant characters.
- Hidden Representation Mix Layer: Within a word, characters exchange information. A "key character"—the one most similar to the word's overall meaning—passes information to others, strengthening the internal word structure.
- Masked Multi-Head Attention and Fusion: The model uses parallel attention modules. One is standard, while the other masks "unimportant" characters to force the model to focus on the semantic core of each word. The outputs are then fused into a new hidden representation.
Experimental Evaluation
The model was evaluated on four diverse public benchmarks: ChnSentiCorp (Sentiment Classification), LCQMC (Sentence Pair Matching), XNLI (Natural Language Inference), and DRCD (Machine Reading Comprehension).
Key Results
The HRMF method provided a consistent performance lift to major baselines, including BERT, BERT-wwm, and ERNIE.
- Sentiment Analysis: The model achieved its most significant gains here (e.g., +0.80 over ERNIE), likely because sentiment is often tied to specific, high-impact words.
- State-of-the-Art Performance: The ERNIE+HRMF combination reached 96.12% accuracy on ChnSentiCorp and 88.29% on LCQMC.
- Ablation Study: Removing the "Mix" layer (HRML) caused the largest drop in performance, proving that the internal exchange of information between characters is the most critical component of the architecture.
Discussion and Conclusion
The HRMF framework demonstrates that characters in a multi-character word are not created equal. By weighting the integration of word semantics and allowing for intra-word communication, the model simulates how humans perceive words as cohesive units. The ensemble "voting" approach for segmentation further stabilizes performance against common preprocessing errors.
Future work will explore the use of dynamic or contextualized word embeddings and the incorporation of additional linguistic features unique to Chinese, such as radicals and Pinyin, to further enrich character representations.
Summary updated