Joint Learning of Chinese Words, Terms and Keywords

Ziqiang Cao, Sujian Li, Heng JiEMNLP 2014 · OpenReview

Abstract

Ziqiang Cao, Sujian Li, Heng Ji. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP).

  1. This paper introduces DP-4, a four-level Dirichlet Process model that jointly performs Chinese word segmentation (WS), term extraction (TE), and keyword extraction (KE). By breaking the traditional "pipeline" approach, the model allows domain-specific and document-level information to improve segmentation accuracy, effectively mitigating error propagation.

Introduction

In Chinese Natural Language Processing, word segmentation (WS) is the foundational first step because the language lacks explicit word boundaries like spaces. Traditionally, researchers have employed a pipelined framework: first segmenting text into words, then identifying domain-specific "terms" (e.g., thrombocytopenia), and finally extracting document-specific "keywords" (e.g., heparinoid).

However, this sequential approach suffers from two major flaws:

  1. Error Propagation: Errors in the initial segmentation phase inevitably cascade, causing subsequent term and keyword extraction to fail.
  2. Information Silos: Pipelined models cannot leverage "downstream" information to improve "upstream" tasks. For instance, knowing a character sequence is a frequent domain-specific term should inform the segmenter to keep it intact, but standard segmenters lack access to this domain-level context.

Methodology: The DP-4 Model

To address these issues, the authors propose a four-level Dirichlet Process (DP-4) model. This Bayesian non-parametric approach captures word distributions across different granularities:

  • Levels 1 & 2 (Corpus): Capture general language patterns across the entire dataset.
  • Level 3 (Domain): Capture distributions specific to a field (e.g., Physics vs. Medicine).
  • Level 4 (Document): Capture distributions specific to a single text.
The graphical representation of the four-level Dirichlet Process (DP-4) model, illustrating the hierarchical dependencies between corpus, domain, and document levels.

Technical Innovations

  • Joint Learning: Words, terms, and keywords are treated as entities that emerge simultaneously during the sampling process, rather than through sequential steps.
  • Sentence-wise Gibbs Sampler: To maintain computational feasibility, the authors developed a sampler that processes entire sentences using the Viterbi algorithm, significantly improving efficiency over character-by-character sampling.
  • Scoring Functions: The model identifies terms and keywords by comparing the probability of a word appearing in a specific domain or document against its probability in the general corpus. Words significantly more likely in a specific context are flagged accordingly.

Experimental Setup

The model was evaluated using two primary data sources:

  1. PKU Corpus: A standard dataset of manually segmented news texts (approx. 1.87 million characters) used to train base corpus-level distributions.
  2. New 10-Domain Corpus: A newly collected dataset of 2,000 documents across ten domains (e.g., Physics, Medicine, Art, Economy). Each document was manually annotated for gold-standard segmentation, terms, and keywords.

Results and Discussion

The DP-4 model was compared against standard baselines, including Conditional Random Fields (CRFs) and the Hierarchical Dirichlet Process (HDP).

Performance comparison of DP-4 against baseline methods across word segmentation (F1), out-of-vocabulary recovery (OOV-R), term extraction (TE), and keyword extraction (KE) tasks.

As shown in the results table, DP-4 consistently outperformed existing methods:

  • Word Segmentation: DP-4 achieved an F1 score of 0.821, surpassing the state-of-the-art CRF (0.817).
  • Term/Keyword Extraction: The improvements were most pronounced here. For Term Extraction (Top 50), DP-4 reached an accuracy of 0.704, compared to the CRF-pipeline's 0.624.
  • Out-of-Vocabulary (OOV) Recovery: DP-4 demonstrated a superior ability to handle unseen words (OOV-R of 0.374 vs. CRF's 0.330), proving that joint learning helps the model "discover" new technical terms through contextual feedback.

Conclusion

The DP-4 model demonstrates that joint learning effectively mitigates error propagation in Chinese NLP. By modeling terms and keywords as deviations from a base language distribution, the system creates a virtuous cycle where better term recognition leads to more accurate segmentation. Future work may involve incorporating Part-of-Speech (POS) tags to further refine results and optimizing the model's computational complexity.

Summary updated

BibTeX
@inproceedings{cao2014joint,
  author = {Ziqiang Cao and Sujian Li and Heng Ji},
  title = {{Joint Learning of Chinese Words, Terms and Keywords}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2014},
  url = {https://openreview.net/forum?id=qdmsOpdKtLc},
}
Figure