TL;DR
This paper introduces a novel "direct text scanning" method for analyzing the complex structure of Japanese compound nouns, which often lack clear word boundaries and contain unregistered words. By using a corpus-driven approach with pattern matchers and a bootstrapping mechanism, the method effectively corrects initial segmentation errors and achieves higher accuracy than traditional methods.
What Problem Does This Paper Address?
Japanese compound nouns, particularly those found in newspaper articles, present a significant challenge for natural language processing. These nouns can be exceptionally long (up to ten constituent words, or twenty kanji characters), lack explicit word delimiters, and frequently incorporate unregistered words like abbreviations. Traditional morphological analyzers struggle with these characteristics, leading to over-segmentation errors. Existing methods for analyzing compound noun structure often rely on costly parsing or brittle conceptual dependency models that fail when encountering unregistered words. This paper aims to develop a practical and accurate method for analyzing the word dependency structure of these complex Japanese compound nouns, specifically addressing the issues of segmentation ambiguity and unregistered words.
Methods
The core of the proposed method is Direct Text Scanning. It begins by using a standard Japanese morphological analyzer (JMA) for an initial segmentation of the compound noun. This initial segmentation is prone to errors, especially with unregistered words. The key innovation lies in a bootstrapping procedure:
- Pattern Matching: The system employs a set of 23 pattern matchers to scan a large corpus for co-occurrence evidence between words. These patterns are designed to capture various types of relationships (e.g., inner-word collocation, particle-combined collocation, adjectival modification, predicate-argument relations).
- Evidence Database (EDB): The gathered co-occurrence evidence is stored in an EDB.
- Feedback Loop: Crucially, newly identified unregistered words are added back to the list of words to be analyzed (WORD_LIST). This feedback loop allows the system to iteratively refine its understanding and gather more evidence for these previously unknown words.
- Heuristics for Unregistered Words: Specific rules (R-(a) through R-(d)) are developed to identify and register potential unregistered words based on their length, context, and dictionary status.
- Dependency Structure Analysis: After the bootstrapping process, the compound noun is re-analyzed with the newly registered words. A parser then evaluates possible dependency structures, using mutual information and cost calculation to select the most plausible one.
The paper also introduces a baseline method based on "leftmost derivation" for comparison.
Datasets
- Corpus: The experiments were conducted using newspaper articles from the "Nikkei Shinbun" for January and February 1992. This corpus contained approximately 27,000 articles, totaling about 7 million characters.
- Test Data: A set of 400 compound nouns was used for evaluation, with 100 samples each for lengths 5, 6, 7, and 8 kanji characters. These were confirmed to be compound nouns.
- Benchmark: A baseline method (leftmost derivation) was introduced and used for comparison.
Key Insights and Ideas
- Bootstrapping for Unregistered Words: The most significant insight is the effectiveness of the bootstrapping approach. By iteratively registering and re-analyzing unregistered words, the system can overcome initial segmentation errors and significantly improve accuracy.
- Corpus-Driven Evidence: Relying on direct text scanning and co-occurrence evidence from a large corpus makes the method adaptable and less dependent on a comprehensive, pre-existing dictionary.
- Handling Ambiguity: The method directly tackles the inherent ambiguity in Japanese compound nouns by gathering external evidence to disambiguate structures.
- Portability: The approach is noted for its portability, as it relies less on the specifics of a morphological analyzer's dictionary and treats registered and unregistered words similarly.
Key Results
- Superior Accuracy: The proposed direct text scanning method significantly outperformed the baseline method across all tested word lengths (5, 6, 7, and 8 kanji). For instance, for 8-kanji words, the direct scanning method achieved an accuracy of 58%, compared to 39% for the baseline.
- Effectiveness on Longer Nouns: The accuracy gap between the proposed method and the baseline widened for longer compound nouns, highlighting the method's strength in handling complexity.
- Error Analysis: The primary causes of errors were identified as proper nouns (42%), time expressions (16%), and monetary expressions (15%).
Limitations and Future Work
- Conceptual Dependency Model: The paper acknowledges that it did not employ a conceptual dependency model, and suggests investigating methods for combining it with the direct text scanning approach.
- Proper Noun Pre-processing: The authors note that proper nouns are a major source of errors and propose that a proper noun pre-processing module should be integrated.
- Corpus Size: The effect of varying corpus size on performance should be investigated.
- Evidence Distance: The influence of the distance between a compound noun and its supporting evidence on cost calculation needs to be explored.
- Parallel Search: To speed up processing, parallel search strategies should be employed.
- Expression Expansion: The paper suggests investigating how to obtain expanded, more natural Japanese sentences from analyzed compound nouns, noting that the
mod-relattribute is currently unused.
Summary updated