Abstract
Sindhi word segmentation is a challenging task due to space omission and insertion issues. The Sindhi language itself adds to this complexity. It's cursive and consists of characters with inherent joining and non-joining properties, independent of word boundaries. Existing Sindhi word segmentation methods rely on designing and combining hand-crafted features. However, these methods have limitations, such as difficulty handling out-of-vocabulary words, limited robustness for other languages, and inefficiency with large amounts of noisy or raw text. Neural network-based models, in contrast, can automatically capture word boundary information without requiring prior knowledge. In this paper, we propose a Subword-Guided Neural Word Segmenter (SGNWS) that addresses word segmentation as a sequence labeling task. The SGNWS model incorporates subword representation learning through a bidirectional long short-term memory encoder, position-aware self-attention, and a conditional random field. Our empirical results demonstrate that the SGNWS model achieves state-of-the-art performance in Sindhi word segmentation on six datasets.
Introduction: The Challenge of Sindhi Word Segmentation
Word segmentation—identifying where one word ends and the next begins—is a foundational step for Natural Language Processing (NLP). While straightforward in languages like English where spaces are consistent delimiters, it is notoriously difficult in Sindhi.
Sindhi uses a cursive Arabic-based script where characters have inherent "joining" or "non-joining" properties. This leads to two major issues:
- Space Omission: When a word ends with a "non-joiner" character, it doesn't physically connect to the next word even without a space. Writers often omit the space, making multiple words look like one to a computer.
- Space Insertion: Conversely, compound words or specific grammatical forms sometimes require a "hard space" to prevent incorrect character joining, which can confuse standard segmenters.
Previous solutions relied on rigid, hand-crafted rules that struggled with "Out-of-Vocabulary" (OOV) words and noisy, real-world text.
Proposed Method: Subword-Guided Neural Word Segmenter (SGNWS)
The authors propose the SGNWS model, which treats segmentation as a sequence labeling task. Instead of just looking for spaces, the model assigns a tag to every character: B (Beginning), I (Inside), E (Ending), S (Single-character word), or X (Hard space).
The architecture consists of four main components:
- Subword Representation Layer (SRL): Instead of just looking at single characters, the model looks at character unigrams, bigrams, and trigrams. This helps the model "see" common prefixes and suffixes, effectively learning the "shape" of Sindhi morphemes.
- BiLSTM Encoder: A bidirectional Long Short-Term Memory network processes the text in both directions, capturing the context of a character from both preceding and succeeding characters.
- Position-Aware Self-Attention: This layer allows the model to weigh the importance of different parts of the input sequence regardless of distance, helping it capture long-range dependencies and remember the relative location of characters.
- CRF Decoder: A Conditional Random Field (CRF) layer ensures that the predicted sequence of tags is linguistically valid (e.g., ensuring an "Inside" tag follows a "Beginning" or "Inside" tag).
Key Results and Insights
The SGNWS model significantly outperformed standard BiLSTM and BiLSTM-CRF baselines across all metrics.
- State-of-the-Art Performance: It achieved an overall F1-score of 98.51% on the combined SDSEG dataset.
- Robustness to Noise: Even on informal data like Twitter, it maintained a high F1-score of 97.24%, demonstrating its ability to handle real-world linguistic variation.
- Handling OOV Words: By incorporating subword features (n-grams), the model effectively handles words it has never seen before, overcoming a major limitation of previous rule-based systems.
Conclusion and Future Work
SGNWS represents a significant advancement in Sindhi NLP. While highly effective, the authors note that non-segmental multiword tokens (like dates or complex numerical expressions) still require additional processing steps. Future work will focus on integrating this segmenter into higher-level tasks like Part-of-Speech (POS) tagging and Named Entity Recognition (NER), as well as exploring more compact subword units for extremely low-resource dialects.
Summary updated