Neural Networks Incorporating Dictionaries for Chinese Word Segmentation

Qi Zhang, Xiaoyu Liu, Jinlan Fu

Abstract

Chinese Word Segmentation (CWS) is a fundamental task in Chinese Natural Language Processing. While neural models have achieved high performance, they often struggle with out-of-vocabulary (OOV) words and domain-specific terminology. This paper explores two neural architectures that integrate external dictionary information to address these challenges. Specifically, we introduce a HyperLSTM-based approach that dynamically adjusts model weights based on dictionary entries, leading to significant improvements in both in-domain and cross-domain segmentation tasks.

Introduction

Unlike many Western languages, Chinese text lacks explicit word boundaries. Consequently, Chinese Word Segmentation (CWS) is the essential first step for downstream tasks such as machine translation and sentiment analysis. Modern CWS systems typically employ Bi-LSTM-CRF architectures. However, these models face two primary hurdles: the OOV problem, where rare words not seen during training are mis-segmented, and domain shift, where models trained on general news corpora fail on specialized technical or literary texts.

Examples of rare and domain-specific words that pose challenges for standard CWS models.

External dictionaries provide a wealth of prior knowledge that can mitigate these issues. This research proposes novel methods to incorporate this static knowledge into dynamic neural networks, treating dictionary matches as helpful hints rather than rigid rules.

Methodology

The core of the proposed approach involves converting dictionary information into a format compatible with neural networks and then using that information to guide the segmentation process.

Feature Vector Construction

For each character in a sentence, the system checks surrounding n-grams (sequences of 2 to 5 characters) against an external dictionary. This lookup process generates an 8-dimensional binary feature vector for every character, representing the presence or absence of dictionary matches at various positions relative to the character.

An 8-dimensional binary feature vector representing dictionary lookup results for a character.

Model Architectures

Two models were developed to utilize these feature vectors:

  1. Model-I (Concatenation): This baseline approach uses two parallel Bi-LSTMs—one for character embeddings and one for dictionary feature vectors. The hidden states are concatenated before being passed to a Conditional Random Field (CRF) layer to predict word boundaries.
  2. Model-II (HyperLSTM): This more advanced architecture uses a HyperLSTM to process dictionary features. Instead of contributing directly to the hidden state, the HyperLSTM dynamically generates the weights for the primary "Main LSTM." This allows the model to alter its internal logic based on the dictionary context, effectively relaxing the weight-sharing constraint of standard LSTMs. If a dictionary indicates a character is likely the start of a rare term, the HyperLSTM modifies the Main LSTM's weights to be more sensitive to that boundary.

Experiments and Results

The models were evaluated across nine datasets, including SIGHAN 2005 and CTB6 for in-domain testing, and SIGHAN 2010 for cross-domain evaluation (covering Literature, Computer Science, Medicine, and Finance).

The results demonstrate that Model-II consistently outperforms both Model-I and previous state-of-the-art neural models. The impact is most pronounced in cross-domain scenarios. For instance, in the Computer Science domain, the F1 score improved from 92.65 to 94.70 simply by incorporating a domain-specific dictionary. This highlights the "plug-and-play" capability of the architecture for domain adaptation, as performance can be improved by swapping dictionaries without extensive retraining.

Conclusion

Integrating external dictionaries via HyperLSTMs provides a robust mechanism for improving CWS. By treating dictionary entries as soft constraints and dynamically adjusting model weights, the proposed system handles rare and domain-specific words with significantly higher accuracy. Future work will explore more complex feature engineering, such as including word frequency or part-of-speech tags, and the application of this framework to other sequence labeling tasks like Named Entity Recognition (NER).

Summary updated

Abstract

In recent years, deep neural networks have achieved significant success in Chinese word segmentation and many other natural language processing tasks. Most of these algorithms are end-to-end trainable systems and can effectively process and learn from large scale labeled datasets. However, these methods typically lack the capability of processing rare words and data whose domains are different from training data. Previous statistical methods have demonstrated that human knowledge can provide valuable information for handling rare cases and domain shifting problems. In this paper, we seek to address the problem of incorporating dictionaries into neural networks for the Chinese word segmentation task. Two different methods that extend the bi-directional long short-term memory neural network are proposed to perform the task. To evaluate the performance of the proposed methods, state-of-the-art supervised models based methods and domain adaptation approaches are compared with our methods on nine datasets from different domains. The experimental results demonstrate that the proposed methods can achieve better performance than other state-of-the-art neural network methods and domain adaptation approaches in most cases.
BibTeX
@inproceedings{zhang2018neural,
  author = {Qi Zhang and Xiaoyu Liu and Jinlan Fu},
  title = {{Neural Networks Incorporating Dictionaries for Chinese Word Segmentation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.11959},
}
Figure