TL;DR
This paper introduces a novel multi-language Named-Entity Recognition (NER) system based on Hidden Markov Models (HMMs) that can handle Japanese, Chinese, Korean, and English. The system boasts a common analytical engine adaptable to new languages with just updated lexical rules and language models, and it achieves state-of-the-art accuracy with significantly faster processing speeds compared to Support Vector Machines (SVMs).
What Problem Does This Paper Address?
The explosion of information on the World Wide Web has created a growing demand for cross-language information retrieval. However, a significant portion of this information is not in English, posing a barrier for non-fluent speakers. To bridge this gap, robust multi-language text analysis techniques like morphological analysis and Named-Entity Recognition (NER) are crucial. While previous NER systems have focused on European languages, there was a clear need for a system capable of handling Asian languages, which present unique challenges due to their orthography and word segmentation ambiguities. This paper addresses this need by presenting a unified HMM-based NER system designed for multi-language information retrieval.
Methods
The core of the system is a multi-language NER engine built upon Hidden Markov Models (HMMs). The architecture is designed to be language-independent where possible, with language-specific components handled by a character code converter and a lexical analyzer.
- Character Code Conversion: This component handles the conversion of various local character encodings (like EUC-JP, EUC-CN, EUC-KR) to Unicode, ensuring a unified internal representation.
- Lexical Analyzer: This module is responsible for identifying word candidates within a sentence. It uses regular expressions and language-specific rules to account for differences in character types, word length, orthography, and spacing. A key innovation here is the ability to generate word candidates for unknown words, crucial for languages without explicit word delimiters.
- Analytical Engine: This language-independent component comprises an N-best word sequence search and a statistical language model (specifically, a word bigram model for morphological analysis and an HMM for NER). The system uses a forward-DP backward-A* N-best search algorithm to handle morpheme sequence ambiguity, a significant advantage for Asian languages.
- Named-Entity Recognition Model: The paper adapts the BBN's HMM for NER. To improve accuracy, especially for languages with word segmentation ambiguity, they incorporate morphological analysis prior to NER, generating a word graph of morpheme candidates. They also extend the linear interpolation scheme for probability estimation, incorporating part-of-speech frequency alongside morpheme frequency.
Datasets
The authors prepared original corpora for Japanese, Chinese, Korean, and English, primarily sourced from newspapers and web texts. They utilized existing morpheme analysis definitions for English (Pen Tree Bank), Japanese (Jtag), Chinese (Beijing Univ.), and Korean (MATEC99). Named-entity tag definitions were based on MUC for English and IREX for Japanese, with Chinese and Korean tags following Japanese IREX specifications. For the bilingual lexicon extraction experiment, they used Japanese-English news article alignment data released by CRL, separating it into a small set (approx. 1000 sentences) for lexicon extraction and a large set (approx. 150,000 sentences) for estimating translation probabilities.
Key Insights and Ideas
- Unified Architecture for Diverse Languages: The system's strength lies in its common analytical engine, making it adaptable to new languages by simply swapping out lexical rules and language models. This is a significant step towards a truly universal NER system.
- Handling Asian Language Ambiguities: The paper effectively tackles the challenges of Asian languages, particularly word segmentation ambiguity, by integrating morphological analysis and N-best morpheme sequence candidates. This approach mitigates the problem rather than relying solely on a single segmentation.
- Speed vs. Accuracy Trade-off: While HMMs are generally considered less accurate than SVMs for NER, this paper demonstrates that their HMM-based system achieves comparable accuracy with a substantial speed advantage (ten times faster than SVM). This is crucial for real-time applications like information retrieval.
- Bilingual Lexicon Extraction: The application of their NER system to extract bilingual lexicons from parallel texts shows its practical utility beyond just recognition, offering a method for automatically building valuable linguistic resources.
Key Results
- The system achieved state-of-the-art accuracy in four languages (English, Japanese, Chinese, Korean).
- The HMM-based system was ten times faster than SVMs for analysis.
- F-measure scores for the HMM system were: EN (88.2%), JP (81.0%), CN (84.5%), KR (79.9%).
- The bilingual lexicon extraction experiment yielded high-scoring candidates, with an estimated accuracy of about 86% for the top 50 candidates.
Limitations and Future Work
- SVM Accuracy for Japanese: The authors noted an unusually lower accuracy for SVM on Japanese data compared to HMM. They hypothesize this might be due to SVM's inability to handle N-best morpheme candidates and its potential unsuitability for irregular, ill-structured, or informal text like web content.
- Detailed Evaluation of Bilingual Lexicons: While the bilingual lexicon extraction showed promise, a more detailed evaluation of the extracted lexicons is acknowledged as a future task.
- Further Analysis of Results: The authors state that a more detailed analysis of the named-entity recognition results is a next step.
Summary updated