TL;DR
This paper introduces a novel statistical method for segmenting Japanese text, specifically focusing on kanji-only sequences, which are notoriously difficult for traditional morphological analyzers. By leveraging large amounts of unsegmented training data and character n-gram statistics, their approach achieves performance comparable to, and sometimes surpassing, existing state-of-the-art methods, all while requiring minimal human-annotated data.
What Problem Does This Paper Address?
Written Japanese lacks explicit word delimiters like spaces, making word segmentation a critical preprocessing step for many natural language processing tasks. Traditional methods often rely on extensive lexicons and grammars, which are costly to build and maintain. A particular challenge arises with sequences composed solely of kanji characters. These sequences often contain domain-specific terms and proper nouns that are absent from general-purpose lexicons, leading to an "unknown word problem." Furthermore, kanji-only sequences often represent compound nouns where grammatical cues are insufficient for disambiguation, and character type changes (a common heuristic in Japanese segmentation) are not applicable. This paper aims to address the segmentation of these challenging kanji sequences using a statistical approach that minimizes reliance on human-curated linguistic resources.
Methods
The core of the proposed method lies in analyzing character n-gram frequencies within a large, unsegmented corpus. For a given potential word boundary between two characters, the algorithm assesses the likelihood of a boundary by comparing the frequencies of n-grams that do not span the boundary (i.e., are entirely to the left or right) against the frequencies of n-grams that do span the boundary. If non-spanning n-grams are significantly more frequent, it suggests a word boundary.
Specifically, the algorithm defines a set of n-gram orders, $N$, to consider. For each location $k$ in the text, it calculates a score $v_n(k)$ for each $n \in N$. This score is derived from the counts of non-straddling n-grams ($s_1, s_2$) and straddling n-grams ($t_j$) in the training corpus. The overall score for a location $k$, $V_N(k)$, is the average of these $v_n(k)$ scores across all $n \in N$. Word boundaries are then placed at locations $k$ where $V_N(k)$ is a local maximum or where $V_N(k)$ exceeds a predefined threshold $t$. This threshold helps in identifying single-character words and controls the granularity of segmentation. The "mostly-unsupervised" aspect comes from the fact that only a very small set of pre-segmented examples are needed to tune the parameters $N$ and $t$.
Datasets
The experiments were conducted using 150 megabytes of 1993 Nikkei newswire data. This data was used to derive character n-gram counts from the unsegmented corpus. For parameter training and evaluation, five held-out subsets of 500 kanji sequences each were created. These sequences were hand-segmented to serve as gold-standard annotations. Each held-out subset was further split into a 50-sequence parameter-training set and a 450-sequence test set.
Key Insights and Ideas
- Leveraging Unsegmented Data: The most significant insight is the demonstration that a purely statistical approach, relying heavily on the statistical properties of character sequences in large unsegmented corpora, can achieve high performance without extensive linguistic resources.
- Focus on Kanji Sequences: The paper highlights the specific challenges of kanji-only sequences and provides a method tailored to overcome them, where traditional methods often falter.
- N-gram Frequency as a Boundary Indicator: The core idea of comparing spanning vs. non-spanning n-gram frequencies is a clever way to infer word boundaries based on cohesion within potential words versus cohesion across potential boundaries.
- Minimal Supervision is Sufficient: The "mostly-unsupervised" claim is substantiated by showing that very few (as few as 5) hand-segmented examples are enough to tune parameters for good performance, significantly reducing the annotation burden compared to supervised methods.
- Novel Evaluation Metrics: The introduction of "compatible brackets" and "all-compatible brackets" rates provides a more robust evaluation, accounting for different segmentation granularities and the potential for errors that are not strictly crossing or morpheme-dividing.
Key Results
- The proposed algorithm achieved performance comparable to, and in some cases superior to, state-of-the-art morphological analyzers like Chasen and Juman.
- Specifically, the algorithm achieved 5.27% higher word precision and 0.26% better F-measure accuracy than Juman, and even better results compared to Chasen.
- For morpheme accuracy, the algorithm showed better recall (by 6.51%) and F-measure (by 1.38%) than Juman, and significantly better results than Chasen.
- The method demonstrated robustness to the size of the training set, with performance not significantly degraded even when using only five annotated sequences.
- The novel compatible bracket metrics showed the algorithm performing better than Chasen and Juman, indicating its ability to produce segmentations that are more "compatible" with human annotations across different levels of granularity.
Limitations and Future Work
- Limited to Kanji Sequences: The current work focuses specifically on kanji sequences. The authors plan to experiment with Japanese sentences containing mixtures of character types (kanji, hiragana, katakana) in future work.
- Parameter Tuning: While minimal, some pre-segmented data is still required for parameter tuning ($N$ and $t$). The authors deem this "mostly unsupervised" due to the small amount of data needed.
- Potential for Japanese-Dependent Heuristics: Although the core method is designed to be language-independent, the evaluation was performed on Japanese data. The authors express interest in testing the method on Chinese or other languages to further validate its portability.
- Balancing Strengths: Future work may involve combining their statistical method with morphological analyzers to leverage the strengths of both approaches.
Summary updated