Introduction
Morphological analysis—the process of segmenting text into words and assigning parts of speech (POS)—is the foundational step for Japanese Natural Language Processing (NLP). Unlike English, Japanese does not use white spaces to delimit words, making segmentation a complex task. While traditional dictionary-based analyzers are highly accurate on formal text like newspapers, they frequently fail when encountering the "wild" language of the web.
Web text is characterized by rapidly evolving slang (e.g., gugu-ru for "to google"), informal spellings, and niche technical terms. When an analyzer encounters an unknown word, it often missegments the entire sentence, leading to downstream errors. Traditional solutions rely on "batch" processing—collecting massive amounts of text and running extraction algorithms offline. This approach is inefficient for real-time adaptation to specific topics or emerging trends. This paper proposes an online lexicon acquisition system that identifies and learns new morphemes in real-time as it processes text.
System Architecture
The proposed system operates as an online lexicon acquirer that works in a tight loop with a morphological analyzer (JUMAN) and a dependency parser (KNP). The architecture is designed to update its internal dictionary dynamically, allowing the system to become more "intelligent" as it reads more data.
As shown in the architecture diagram, the acquisition process follows four main phases:
- Detection: As the analyzer processes a sentence, it flags sequences it cannot identify using a special "undefined" tag.
- Candidate Enumeration: The system examines the surrounding context of the undefined sequence to hypothesize potential word boundaries and parts of speech.
- Aggregation: Candidates are stored in a "trie" structure. Every time a similar-looking unknown sequence appears, the system adds it to the accumulated examples.
- Selection: Once sufficient evidence is gathered, the system compares the different examples. If a specific stem and POS tag consistently explain all observed instances, the word is "promoted" to the automatically constructed dictionary.
Leveraging Morphological Constraints
The core innovation of this system is the use of Japanese morphological constraints to reduce the search space for unknown words. Because Japanese is an agglutinative language, a word's stem is followed by specific suffixes or particles that dictate its grammatical function.
The authors pre-compiled a library of 500,000 unique suffixes from the web. By matching the characters following an unknown word against this library, the system can hypothesize the word's stem and its likely POS tag almost immediately. For example, seeing a word followed by "-tte mita" (a common verbal suffix) strongly suggests the preceding characters form a verb stem. This allows the system to learn new words with as few as four to nine examples, far fewer than statistical methods typically require.
Experimental Results
The system was evaluated using domain-specific corpora crawled from the web (e.g., "Whaling issue," "Tsundere," "JASRAC"). The initial "seed" dictionary was the standard JUMAN dictionary containing roughly 120,000 morphemes.
Key findings include:
- High Precision: The system acquired new morphemes with an accuracy between 93.9% and 99.3% across different domains.
- Efficiency: The median number of examples required to "learn" a new word was remarkably low (4 to 9 occurrences).
- Vocabulary Growth: The system showed a steady, monotonic increase in vocabulary. For the "JASRAC" query, nearly 1,000 new morphemes were acquired after processing 200,000 sentences.
- Improved Analysis: Manual evaluation of "diff" blocks confirmed that the overwhelming majority of changes introduced by the new dictionary were corrections, moving the analysis from "Erroneous" to "Correct."
Conclusion and Future Work
By using grammatical constraints as shortcuts, the proposed system effectively "seizes the head" of the unknown word distribution, cleaning up the most impactful errors quickly. However, challenges remain, such as the oversegmentation of long Katakana strings and detection gaps where the analyzer incorrectly guesses an unknown word is a combination of known words. Future work will focus on more comprehensive detection methods that look for statistical anomalies in analyzer confidence rather than relying solely on "undefined" tags.
Summary updated