A Subword Guided Neural Word Segmentation Model for Sindhi

Wazir Ali, Jay Kumar, Zenglin Xu, Congjian Luo, Junyu Lu, Junming Shao, Rajesh Kumar, Yazhou Ren

Abstract

Deep neural networks employ multiple processing layers for learning text representations to alleviate the burden of manual feature engineering in Natural Language Processing (NLP). Such text representations are widely used to extract features from unlabeled data. Word segmentation is a fundamental and inevitable prerequisite for many languages. Sindhi is an under-resourced language whose segmentation is challenging as it exhibits space omission and space insertion issues and lacks a labeled corpus for segmentation. In this paper, we investigate supervised Sindhi Word Segmentation (SWS) using unlabeled data with a Subword Guided Neural Word Segmenter (SGNWS) for Sindhi. To learn text representations, we incorporate subword representations into a recurrent neural architecture to capture word information at the morphemic level, taking advantage of Bidirectional Long-Short Term Memory (BiLSTM), a self-attention mechanism, and a Conditional Random Field (CRF). Our proposed SGNWS model achieves an F1 value of 98.51% without relying on feature engineering. The empirical results demonstrate the benefits of the proposed model over existing Sindhi word segmenters.

Introduction

Word segmentation is the foundational step in NLP, identifying word boundaries in continuous text. While trivial in languages like English, it is notoriously difficult in Sindhi. Sindhi uses a Persian-Arabic script that is cursive and contains both "joiner" and "non-joiner" characters. This orthographic property leads to two major challenges: space omission and space insertion.

Space omission occurs because certain characters do not naturally join with the following letter, leading writers to omit spaces between words while maintaining visual readability. Conversely, space insertion occurs when spaces are added within a single word (such as compound words or abbreviations) to prevent incorrect character joining. Traditional rule-based systems often fail to handle these nuances, especially with out-of-vocabulary (OOV) words or noisy social media text.

Table showing different Sindhi word types and the resulting space omission/insertion challenges.

Methodology: The SGNWS Architecture

The proposed Subword-Guided Neural Word Segmenter (SGNWS) treats segmentation as a sequence labeling task using a fine-grained [B, I, E, S, X] tagging scheme (Beginning, Inside, End, Single, and Hard Space). The architecture consists of four primary layers designed to automate feature extraction and capture both local and global context.

The SGNWS architecture, illustrating the flow from subword sequences through BiLSTM and self-attention layers to the CRF decoder.

1. Subword Representation Layer (SRL)

Instead of relying solely on individual characters, the model utilizes character unigrams, bigrams, and trigrams. This subword approach allows the model to capture morphemic-level information, which is critical for understanding the internal structure of Sindhi words and improving performance on rare or unseen words.

2. BiLSTM Encoder

A Bidirectional Long Short-Term Memory (BiLSTM) network processes the input sequence in both forward and backward directions. This ensures that the representation of each character is informed by its entire surrounding context within the sentence.

3. Position-Aware Self-Attention

To capture long-range dependencies that a standard LSTM might miss, a self-attention mechanism is integrated. This layer allows the model to weigh the importance of different parts of a sentence dynamically, regardless of their distance from the current character being processed.

4. CRF Decoder

The final layer is a Conditional Random Field (CRF) decoder. The CRF models the dependencies between neighboring labels, ensuring that the predicted tag sequence is linguistically valid (e.g., preventing an "Inside" tag from following a "Space" tag).

Experimental Results

The SGNWS model was evaluated on a comprehensive dataset (SDSEG) comprising over 2.3 million tokens from diverse sources, including news (Kawish, Awami Awaz), Wikipedia, books, and Twitter.

Key findings include:

  • State-of-the-Art Performance: The model achieved an overall F1-score of 98.51%, significantly outperforming standard BiLSTM and BiLSTM-CRF baselines.
  • Domain Robustness: Despite the informal and noisy nature of social media text, the model maintained a high F1-score of 97.24% on Twitter data, demonstrating its ability to handle inconsistent spacing and slang.
  • Feature Efficacy: Ablation studies confirmed that the transition from character-level to subword-level (trigram) features provided the most significant performance boost, highlighting the importance of subword information in morphologically rich languages.

Conclusion

The SGNWS model provides a robust, end-to-end solution for Sindhi word segmentation that eliminates the need for manual feature engineering. By combining subword representations with self-attention and BiLSTM-CRF architectures, the model effectively addresses the unique orthographic challenges of the Sindhi script. Future work will explore the applicability of this architecture to other low-resource languages using similar scripts, such as Urdu and Pashto.

Summary updated

Abstract

Deep neural networks employ multiple processing layers for learning text representations to alleviate the burden of manual feature engineering in Natural Language Processing (NLP). Such text representations are widely used to extract features from unlabeled data. The word segmentation is a fundamental and inevitable prerequisite for many languages. Sindhi is an under-resourced language, whose segmentation is challenging as it exhibits space omission, space insertion issues, and lacks the labeled corpus for segmentation. In this paper, we investigate supervised Sindhi Word Segmentation (SWS) using unlabeled data with a Subword Guided Neural Word Segmenter (SGNWS) for Sindhi. In order to learn text representations, we incorporate subword representations to recurrent neural architecture to capture word information at morphemic-level, which takes advantage of Bidirectional Long-Short Term Memory (BiLSTM), self-attention mechanism, and Conditional Random Field (CRF). Our proposed SGNWS model achieves an F1 value of 98.51% without relying on feature engineering. The empirical results demonstrate the benefits of the proposed model over the existing Sindhi word segmenters.
BibTeX
@inproceedings{ali2020subword,
  author = {Wazir Ali and Jay Kumar and Zenglin Xu and Congjian Luo and Junyu Lu and Junming Shao and Rajesh Kumar and Yazhou Ren},
  title = {{A Subword Guided Neural Word Segmentation Model for Sindhi}},
  booktitle = {IEEE Access},
  year = {2020},
  doi = {10.48550/arxiv.2012.15079},
}
Figure