TL;DR
This paper introduces a statistical method for extracting Chinese compound words from massive corpora, leveraging mutual information and context dependency. Their approach demonstrates high precision and robustness, outperforming previous methods and showing practical utility in improving information retrieval systems.
What Problem Does This Paper Address?
Chinese, unlike many Western languages, lacks explicit word delimiters (spaces). This makes it challenging to identify individual words and, consequently, compound words (words formed from two or more characters). Traditional rule-based methods are often insufficient, and existing statistical approaches have typically been evaluated on relatively small corpora, leaving their effectiveness on large-scale, real-world data uncertain. Furthermore, Chinese is a dynamic language where new compound words are constantly being created, making it impossible to maintain a comprehensive, static lexicon. This necessitates robust automatic methods for compound extraction.
Methods
The authors propose a two-step statistical approach:
- Mutual Information (MI): This step identifies candidate compound words by measuring the correlation between adjacent characters or character sequences. Higher MI values indicate a stronger association, suggesting a potential compound. They apply this to bi-grams, tri-grams, and quad-grams, using a pre-set MI threshold.
- Context Dependency: This step refines the candidate list by ensuring that extracted compounds are "complete" words and not just parts of larger words. It checks for "left context dependency" (LSize, MaxL) and "right context dependency" (RSize, MaxR), essentially measuring how strongly the compound is connected to its surrounding words. Compounds with strong context dependency are filtered out.
The core idea is that a valid compound should have strong internal correlation (high MI) but weak correlation with its immediate neighbors.
Datasets
The experiments were conducted on three corpora:
- Corpus A: A homogeneous corpus of political and economic news (325 million characters).
- Corpus B: A heterogeneous corpus combining novels, technique reports, and news (650 million characters).
- Corpus C: A TREC Chinese corpus containing People's Daily and Xinhua news (75 million characters), used for information retrieval experiments.
All corpora were pre-segmented using an initial lexicon of 65,502 entries.
Key Insights and Ideas
- Large Corpora Matter: The study strongly suggests that using very large corpora significantly improves the quality and quantity of extracted compounds, a finding not clearly demonstrated by prior work on smaller datasets.
- MI vs. Context Dependency Roles: Mutual Information primarily influences the precision of the extracted compounds, while context dependency primarily affects the count of extracted items. This provides a clear understanding of how to tune the parameters for desired outcomes.
- Homogeneity Aids Extraction: The method performs better on homogeneous corpora (like news) than on heterogeneous ones. This is attributed to the more consistent and "well-normalized" nature of text in homogeneous domains.
- Practical Utility in IR: Adding the automatically extracted compounds to an existing lexicon demonstrably improves information retrieval performance, showing the real-world applicability of the method.
Key Results
- The proposed method successfully extracted over 50,000 new compounds from raw texts.
- Experiments on Corpus A (homogeneous) showed high precision (e.g., 96.7% for quad-grams with specific parameters).
- When applied to Corpus B (heterogeneous) with the same parameters, precision dropped significantly (to 88.3%), highlighting the impact of corpus type.
- Tuning parameters on Corpus B showed that stricter context dependency settings led to fewer compounds but higher precision.
- In information retrieval experiments on Corpus C, adding the extracted compounds to the lexicon resulted in a 1.2% average precision improvement.
Limitations and Future Work
- Subjectivity of "Compound": The authors acknowledge that defining a universally accepted definition of a "compound" is difficult, and their evaluation relies on manual checking by a single person to mitigate inter-annotator divergence.
- Parameter Sensitivity: The performance is sensitive to parameter settings, and optimal settings may vary significantly between different corpus types (homogeneous vs. heterogeneous). The paper suggests that using the same parameter settings for different corpora might not be suitable.
- Negative Impact on Some Queries: While generally beneficial, the new lexicon negatively impacted a small number of queries, indicating that not all extracted compounds are perfectly correct and can sometimes lead to mis-segmentation.
- Future Work: The paper implicitly suggests further exploration into adapting parameter settings for different corpus types and investigating methods to further improve the precision of extracted compounds, especially for heterogeneous domains.
Summary updated