Abstract
Word Segmentation (WS) is a foundational task for Natural Language Processing in East Asian languages like Chinese, Japanese, and Korean, where words are not delimited by spaces. Traditional approaches treat WS as a character-level sequence labeling task using schemas like BMES (Begin, Middle, End, Single). However, these schemas introduce complex transition constraints that require sophisticated decoding layers. This paper proposes Separation Inference (SpIn), a framework that redefines WS as a binary classification of the intervals between adjacent characters. By predicting whether a "Connection" (C) or "No-Connection" (NC) exists between characters, SpIn simplifies the task, eliminates illegal tag transitions, and achieves new state-of-the-art results across multiple languages.
Introduction
In languages such as Chinese, Japanese, and Korean (CJK), the lack of explicit word boundaries necessitates a segmentation step before downstream NLP tasks. Traditionally, this is modeled as sequence labeling, where each character is assigned a tag indicating its position within a word. A significant drawback of this approach is the "implicit position information" burden; for instance, in a BMES system, a 'B' tag cannot legally be followed by another 'B'. To manage these constraints, researchers often rely on complex Conditional Random Field (CRF) layers.
The authors of SpIn argue that the exact relative position of a character is less critical than the simple binary state of the boundary between two adjacent characters. By focusing on the intervals rather than the characters themselves, the problem is reduced to a straightforward binary classification task.
The SpIn Framework
Connection-No-Connection (C-NC) Tagging
The core innovation of SpIn is the C-NC tagging schema. Instead of labeling characters, the model labels the bigrams (adjacent character pairs):
- Connection (C): Indicates the two characters belong to the same word.
- No-Connection (NC): Indicates a word boundary exists between the characters.
This schema inherently avoids the "illegal transition" problem. Because any sequence of C and NC tags is linguistically valid at the structural level, the model does not require a complex CRF to enforce tag consistency, although a CRF can still be used to capture local dependencies.
Model Architectures
The SpIn framework is versatile and can be implemented using both traditional machine learning and modern deep learning techniques:
- Machine Learning Approach: Utilizes feature templates (unigrams, bigrams, and symbol types) processed through a CRF. This version is highly efficient, running up to 500 times faster than deep learning models with only a marginal performance trade-off.
- Deep Learning Approach: Employs a BERT-based feature extractor. Character embeddings are projected and concatenated to form bigram features, which are then passed through fully connected layers and a Softmax or CRF layer for inference.
Experimental Results
SpIn was evaluated on major benchmarks including CTB6 and SIGHAN 2005 for Chinese, BCCWJ for Japanese, and KAIST/GSD for Korean.
- State-of-the-Art Performance: SpIn outperformed previous dominant deep learning methods across all five Chinese datasets. Notably, it achieved a +1.08% F1 score improvement on the PKU dataset.
- Out-of-Vocabulary (OOV) Robustness: One of the most significant gains was in OOV recall. On the PKU dataset, OOV recall improved by +6.77%, suggesting that focusing on local connection logic allows the model to generalize better to unseen words than position-based memorization.
- Cross-Language Utility: The framework set new SOTA benchmarks for Japanese and Korean, confirming that the C-NC logic is a universal property of East Asian scripts.
Conclusion
Separation Inference represents a paradigm shift in Word Segmentation by treating bigrams as the natural unit of analysis. By simplifying the tagging schema to a binary Connection/No-Connection decision, SpIn reduces computational overhead, eliminates the need for complex transition constraints, and provides superior generalization for out-of-vocabulary terms. Future work may involve extending this binary inference logic to joint tasks like POS tagging and dependency parsing.
Summary updated