TL;DR
This paper introduces CSeg&Tag1.0, a novel integrated system for Chinese word segmentation and POS tagging that treats these two tasks as a single problem. It achieves high performance on unrestricted texts by employing a multi-agent architecture that effectively handles ambiguities and unknown words, a significant challenge in Chinese NLP.
What Problem Does This Paper Address?
Chinese, unlike many Western languages, lacks explicit word delimiters like spaces. This makes word segmentation (identifying word boundaries) a crucial first step for any downstream NLP task. Following segmentation, each word needs to be assigned its part-of-speech (POS) tag. While significant research has been dedicated to these problems, practical systems with high performance on unrestricted Chinese texts have been elusive. The primary obstacles are:
- Ambiguity: A sequence of characters can often be segmented in multiple valid ways, leading to different meanings.
- Unknown Words: Transliterated foreign names, personal names, and place names are common but not always present in dictionaries, posing a challenge for both segmentation and tagging.
Existing approaches often tackle these issues separately or use simplified integration methods, failing to capture the deep interdependencies between segmentation and tagging.
Methods
CSeg&Tag1.0 is designed as a three-level multi-agent system:
- Low Level Agents: Dedicated agents (CNAgent, TFNAgent, CPNAgent) focus on identifying potential unknown words (Chinese personal names, transliterated foreign names, Chinese place names). They leverage extensive knowledge bases (CN Bank, TFN Bank, CPN Bank) and employ strategies like maximal matching and character set analysis.
- Intermediate Level (Competition Agent): This agent resolves conflicts that arise when multiple low-level agents propose overlapping or contradictory candidates. It uses a comprehensive evaluation function considering various statistical and structural information.
- High Level (Bigram-based Agent): This agent handles the remaining ambiguities by employing a conventional POS bigram model and a dynamic programming algorithm to find the optimal segmentation and tagging sequence.
The system also incorporates mechanisms for "global statistics" (derived from large corpora) and "local statistics" (derived from the current input text, acting as a cache) to improve disambiguation. A rule base containing word formation rules, syntactic rules, and heuristics further aids the process.
Datasets
The paper mentions the following datasets:
- CN Bank (CNB): 200,000 samples of Chinese personal names.
- TFN Bank (TFNB): 38,769 samples of transliterated foreign names.
- CPN Bank (CPNB): 17,637 samples of Chinese place names.
- Dictionary: Contains 60,133 word entries with frequencies and POS information.
- Manually Tagged Corpus: Approximately 0.4 million words used for training the bigram model.
- Raw Corpus: 20 million characters used for achieving global statistics.
Key Insights and Ideas
- Integrated Approach is Crucial: The paper strongly advocates for treating word segmentation and POS tagging as a single, integrated problem rather than sequential steps. This is because decisions made during segmentation directly impact tagging possibilities, and vice-versa.
- Multi-Agent Architecture for Specialization: Breaking down the complex problem into specialized agents (for unknown words, conflict resolution, and final disambiguation) allows for more focused and effective processing.
- Importance of "Knowledge": The authors emphasize that overcoming the "knowledge" gap is key to solving Chinese NLP problems. This includes not just formal algorithms but also how to represent, extract, and cooperatively master various types of knowledge (lexical, statistical, syntactic).
- Local Statistics (Cache) are Highly Valuable: The paper highlights the significant impact of local statistics (cache) in resolving ambiguities, especially when dealing with isolated sentences where multiple interpretations might seem plausible.
Key Results
Preliminary open tests on CSeg&Tag1.0 show impressive performance:
- Segmentation Precision: 98.0% - 99.3%
- POS Tagging Precision: 91.0% - 97.1%
- Unknown Word Recall: 95.0% - 99.0%
- Unknown Word Precision: 87.6% - 95.3%
- Processing Speed: Approximately 100 characters per second on a Pentium 133 PC.
Limitations and Future Work
The authors acknowledge that CSeg&Tag1.0 represents the "first round" of their investigation and that significant work remains to achieve their ultimate goal of approximately 99% segmentation and 95% tagging precision for all running Chinese texts.
Future work includes:
- Algorithm Promotion: Carefully promoting the algorithms, particularly those related to agents and the cache mechanism.
- Knowledge Base Improvement: Enlarging the size of relevant resources (textual corpora, unknown word banks) and refining the lexicon, tagged corpus, and rule base.
- Deeper Discussion: The paper mentions that a more in-depth discussion of global and local statistics, as well as the rule base, will be presented in future work.
Summary updated