TL;DR
This paper introduces QJP, a portable and efficient Japanese parser that achieves high accuracy with a remarkably small dictionary and minimal memory footprint. By simplifying structural ambiguity handling and eschewing semantic analysis, QJP offers a practical solution for integrating advanced natural language processing into applications on resource-constrained systems.
What Problem Does This Paper Address?
Natural language parsers are crucial for many document processing tasks, but existing Japanese parsers often suffer from several drawbacks. They typically require massive dictionaries (tens of thousands of words), complex semantic disambiguation techniques, and substantial computational resources (high memory and CPU). This makes them difficult to integrate into practical applications, especially on less powerful hardware. Furthermore, users often need to maintain specialized dictionaries for specific domains, adding to the burden. The authors aim to create a parser that is portable, fast, robust, and easy to use, overcoming these limitations.
Methods
QJP employs a two-stage analysis process:
- Morphological Analysis: This stage segments Japanese sentences into morphemes and words, assigning part-of-speech tags. Instead of relying on a large dictionary for all words, QJP leverages the unique character types in written Japanese. It uses a small dictionary primarily for functional words (like particles and auxiliary verbs) and employs allocation rules based on character types and sequences to extract candidates for content words (nouns, verb stems, etc.). This significantly reduces the dictionary size.
- Syntactic Analysis (Kakari-uke Analysis): This stage determines the dependency structure between "bunsetsu" (phrasal units). QJP simplifies structural ambiguity by:
- Ignoring Semantic Information: It relies solely on morphological/word and syntactic information, avoiding the complexity and potential inaccuracies of semantic analysis.
- Simple Treatment of Structural Ambiguities: Instead of generating all possible parse trees, QJP generates a "best" syntactic structure that preserves local ambiguities. This approach avoids combinatorial explosions in processing time and memory. If the user corrects a parse, the system recalculates the best set of pairs.
Datasets
The paper mentions using two test-sets for tuning and blind testing:
- Test-set 1: 241 sentences, average length 24.1 words/sentence.
- Test-set 2: 210 sentences, average length 29.5 words/sentence.
The paper does not introduce new benchmarks but evaluates QJP against existing Japanese analysers implicitly through performance metrics.
Key Insights and Ideas
- Leveraging Japanese Character Types for Dictionary Reduction: The insight that distinct character sets (Kanji, Hiragana, Katakana) can cue word segmentation and help identify word types is key to building a small dictionary.
- Pragmatic Simplification of Syntactic Ambiguity: The decision to ignore semantic information and simplify structural ambiguity resolution is a crucial trade-off. This allows for a much faster and more resource-efficient parser without a drastic loss in practical accuracy for many applications.
- Portability and Resource Efficiency: The focus on a small dictionary, minimal memory usage, and fast processing speed makes QJP highly suitable for deployment on various platforms, including PCs and workstations, where resources might be limited.
- No Dictionary Maintenance for New Terms: A significant advantage is that QJP does not require users to maintain additional terms in the dictionary, making it more adaptable to new domains.
Key Results
- Dictionary Size: The QJP dictionary contains about 5,000 morphemes and has a total size of less than 50KB.
- Memory Footprint: Approximately 260KB of memory is required on DOS and 500KB on UNIX.
- Processing Speed: QJP can process over 100 words/sec on an 80486-PC and 700-800 words/sec on a Sun-SS20. A 100-word sentence can be analyzed in less than 1 second on a PC.
- Analysis Accuracy:
- Morpheme/word accuracy: 99.3% (test-set 1), 95.7% (test-set 2).
- Uke bunsetsu accuracy (for bunsetsus other than the last): 95.1% (test-set 1), 90.5% (test-set 2).
- Kakari-uke pair set accuracy: 71.0% (test-set 1), 43.8% (test-set 2).
- For correctly morphologically analyzed sentences of 3-15 bunsetsus: Uke bunsetsu accuracy is 97.3% and 70.5% for kakari-uke pairs.
- Performance Comparison: QJP's file size and memory usage are estimated to be more than ten times better than existing Japanese analysers. Its morphological accuracy is slightly lower than analysers with large dictionaries, but syntactic accuracy is comparable.
Limitations and Future Work
- Limitations:
- The morphological accuracy is slightly lower than existing Japanese morphological analysers that use large-scale dictionaries.
- QJP currently does not segment compound Kanji words of Chinese-origin, leaving this task to the application.
- Future Work:
- The authors plan to implement a function for segmenting compound Kanji words using statistical data and affixes.
- QJP has been utilized for keyword extraction, natural language query, and text reading support, with plans for further applications like information retrieval systems.
- Other researchers are using QJP for purposes such as linguistic data extraction.
Summary updated