Abstract
Japanese text is written without spaces, making word segmentation a fundamental challenge for natural language processing. This paper explores a minimalist probabilistic algorithm that identifies word boundaries without relying on a dictionary or complex hand-coded rules. By leveraging character-type transitions and hiragana bigram frequencies, the system achieves a recall of up to 98.3%. These results suggest that statistical properties of text are highly effective proxies for lexical meaning, offering a robust and computationally efficient alternative to traditional rule-based systems.
Introduction
In contrast to English, where spaces clearly demarcate word boundaries, Japanese is written as a continuous string of characters. Automated processing requires initial segmentation—deciding where one word ends and the next begins. Traditional "rationalist" approaches utilize extensive dictionaries and grammar rules, which are often brittle and labor-intensive to maintain. This study adopts an "empiricist" perspective, investigating whether word boundaries can be identified using only the statistical properties of the text itself.
Methodology
The proposed algorithm categorizes characters into five types: Hiragana, Kanji, Katakana, Punctuation, and Romaji. Segmentation logic follows three primary heuristics:
- Type Transitions: A change in character type (e.g., Katakana to Hiragana) is treated as a word boundary.
- Hiragana Bigrams: For sequences of Hiragana, the algorithm employs a 90x90 bigram frequency array. It compares the probability of character co-occurrence against the probability of a boundary existing between them.
- Kanji-to-Hiragana Logic: To handle Hiragana suffixes (okurigana) or functional particles following Kanji, the algorithm uses frequency ratios to determine if the Hiragana continues the preceding word or starts a new one.
Refinements included treating the object-marker particle wo as a standalone unit and allowing eleven specific Hiragana characters to function as single-character words.
Datasets
The algorithm was evaluated across three distinct corpora to account for varying writing styles:
- Business Corpus: 216 newspaper articles from Yomiuri (approx. 49,000 characters).
- Editorial Corpus: Columns from Asahi Shinbun (1.17 million characters).
- Dictionary Corpus: Data from the Shin-Meikai Kokugo Jiten (2.5 million characters).
The model was trained on 90% of the data to establish frequency tables and tested on the remaining 10%.
Results and Analysis
The minimalist approach proved highly effective. In the Business corpus, the initial model correctly segmented 90.7% of Hiragana strings. With minor refinements in the Editorial corpus, the model achieved a recall of 98.3% and a precision of 94.4%.
The high performance indicates that character-type information alone accounts for approximately 84% of segmentation accuracy, while the probabilistic bigram model successfully addresses the remaining complexity in Hiragana-only strings. Furthermore, the algorithm is computationally efficient, avoiding the overhead of dictionary lookups and recursive rule-checking.
Conclusion
This research demonstrates that a lexicon-free, statistical approach can match or exceed the performance of rule-based Japanese segmenters. By prioritizing consistency over rigid linguistic definitions, the minimalist algorithm provides a scalable solution for large-scale text processing. Future work will address the "infrequent sequence" problem and further refine the handling of dense Kanji strings.
Summary updated