Automatic Syllabification for Manipuri language

Loitongbam Gyanendro Singh, Lenin Laitonjam, Sanasam Ranbir SinghCOLING 2016

TL;DR

This paper tackles the challenge of automatically syllabifying words in Manipuri, a low-resource Indian language. They introduce a novel entropy-based phonotactic segmentation method and demonstrate that combining it with sequence labeling techniques like Conditional Random Fields (CRF) significantly improves accuracy, achieving 98% word accuracy.

What Problem Does This Paper Address?

Syllabification, the process of dividing words into their constituent syllables, is crucial for many natural language processing (NLP) tasks like Text-to-Speech (TTS) and Automatic Speech Recognition (ASR). While research exists for well-resourced Indian languages, Manipuri, a scheduled Indian language, has been largely overlooked due to a lack of readily available data. Existing rule-based methods are often expensive to develop, language-dependent, and struggle with loanwords and heteronyms. This paper aims to bridge this gap by developing data-driven methods for Manipuri syllabification, leveraging a newly created corpus.

Methods

The authors propose three main approaches:

  • Rule-based approach:
    • Baseline System (C*VC* rule): A foundational approach based on general syllable structure rules.
  • Entropy Based Phonotactic Segmentation (EPS): A novel, data-driven rule-based method that uses entropy to estimate the information content of phoneme sequences and determine syllable boundaries. This approach is designed to handle issues like schwa deletion.
  • Sequence Labeling approach: This frames syllabification as a sequence labeling task, where each character in a word is tagged as either a "continuous" character (C) or a "splitting" character (S) to mark syllable boundaries. They investigate two state-of-the-art methods:
  • Conditional Random Field (CRF)
    • Maximum Entropy Markov Model (MEMM)
  • Hybrid approach: This combines the strengths of the EPS method and sequence labeling. The output from the sequence labeling model is fed into the EPS method for verification and potential error correction, aiming to leverage both data-driven learning and linguistic phonotactic structure.

Datasets

  • Manipuri Corpus: A new UTF-8 corpus was generated for this study. It consists of 733 documents with 89,084 words and 26,203 unique words, extracted from two local Manipuri newspapers. The OCR accuracy for this corpus was initially low (52.6%), requiring manual correction.
  • Assamese Dataset: A publicly available Assamese dataset of 17,925 unique words was used to test the language independence of the CRF model.

Key Insights and Ideas

  • Entropy as a Syllable Boundary Indicator: The paper highlights that the entropy of phoneme distributions can effectively capture structural biases and help in determining syllable boundaries, especially in languages with complex phonotactics like Manipuri.
  • Sequence Labeling for Robustness: Framing syllabification as a sequence labeling problem allows models to learn complex dependencies automatically, making them more robust to variations like loanwords and schwa deletion, which are challenging for traditional rule-based systems.
  • Hybrid Approach Synergy: Combining rule-based (EPS) and machine learning (CRF) approaches yields the best results, suggesting that leveraging both linguistic knowledge and data-driven learning is key for optimal performance.
  • Language Independence of CRF: The CRF model demonstrates strong performance on an unrelated language (Assamese), indicating its potential for generalization across different linguistic structures.

Key Results

The proposed methods significantly outperform the baseline rule-based approach:

  • Baseline: 87.5% word accuracy.
  • EPS: Achieved 95.99% word accuracy, an improvement of over 8.5%.
  • CRF: Achieved 97.08% word accuracy, outperforming the baseline by about 9.6%.
  • Hybrid Approach: Achieved the highest word accuracy at 97.58%, a further enhancement over CRF and EPS.

The paper also notes that the proposed methods were effective in handling schwa deletion and loan words.

Limitations and Future Work

  • OCR Accuracy: The initial low accuracy of the Bengali OCR used for corpus generation necessitated manual correction, which is a labor-intensive process.
  • MEMM Performance: MEMM underperformed other methods, potentially due to its susceptibility to label bias problems, which EPS and CRF can overcome by generalizing global context.
  • Future Work: The paper suggests further investigation into the language independence of CRF and exploring more advanced sequence labeling techniques. The authors also mention the potential for using the generated Manipuri corpus for other linguistic studies.

Summary updated

BibTeX
@inproceedings{singh2016automatic,
  author = {Loitongbam Gyanendro Singh and Lenin Laitonjam and Sanasam Ranbir Singh},
  title = {{Automatic Syllabification for Manipuri language}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2016},
}
Figure