Enhancing Sindhi Word Segmentation using Subword Representation Learning and Position-aware Self-attention
Lists
Only you see this
Abstract
Sindhi word segmentation is a challenging task due to space omission and insertion issues. The Sindhi language itself adds to this complexity. It's cursive and consists of characters with inherent joining and non-joining properties, independent of word boundaries. Existing Sindhi word segmentation methods rely on designing and combining hand-crafted features. However, these methods have limitations, such as difficulty handling out-of-vocabulary words, limited robustness for other languages, and inefficiency with large amounts of noisy or raw text. Neural network-based models, in contrast, can automatically capture word boundary information without requiring prior knowledge. In this paper, we propose a Subword-Guided Neural Word Segmenter (SGNWS) that addresses word segmentation as a sequence labeling task. The SGNWS model incorporates subword representation learning through a bidirectional long short-term memory encoder, position-aware self-attention, and a conditional random field. Our empirical results demonstrate that the SGNWS model achieves state-of-the-art performance in Sindhi word segmentation on six datasets.
BibTeX
@inproceedings{ali2020enhancing,
author = {Wazir Ali and Jay Kumar and Saifullah Tumrani and Redhwan Nour and Adeeb Noor and Zenglin Xu},
title = {{Enhancing Sindhi Word Segmentation using Subword Representation Learning and Position-aware Self-attention}},
booktitle = {IEEE Access},
year = {2020},
doi = {10.48550/arxiv.2012.15079},
}Loading related content...