Automatic Conversion of Dialectal Tamil Text to Standard Written Tamil Text using FSTs

Marimuthu Kalimuthu, Sobha Lalitha DeviSIGMORPHON/SIGFSM 2014 · DOI

TL;DR

This paper introduces a novel system, DSWT, that automatically converts informal, dialectal Tamil text into standard written Tamil using Finite State Transducers (FSTs). This is crucial for improving the performance of downstream NLP tasks like machine translation, which often struggle with non-standard language inputs.

What Problem Does This Paper Address?

The internet has democratized content creation, leading to a surge in informal, colloquial, and dialectal language online. While this fosters natural communication, it poses a significant challenge for Natural Language Processing (NLP) systems, which are typically trained on formal, standard language. Tamil, with its over 70 million speakers and numerous dialects, exemplifies this problem. Dialectal variations in pronunciation, grammar, and vocabulary mean that standard NLP tools often fail to process or understand this content accurately. This paper tackles the problem of bridging this gap by creating a system that can transform dialectal and spoken Tamil into its standard written form, thereby making it accessible to existing NLP pipelines.

Methods

The core of the DSWT system relies on a three-pronged approach:

  1. Transformation Engine (FSTs): This component uses Finite State Transducers (FSTs) to model the regular patterns of suffixations and inflections found in spoken Tamil. FSTs are well-suited for this task as they can efficiently map dialectal "root+spoken-language-suffix" forms to "root+written-language-suffix" forms. The paper highlights that processing is done in a right-to-left fashion, which is more efficient given the limited number of unique suffixes compared to root words.
  2. CRF Word Boundary Identifier: Tamil is a morphologically rich language, often featuring agglutination (combining multiple words or morphemes into one) and compounding. To handle these, a Conditional Random Fields (CRF) based word boundary identifier is employed. This module segments agglutinated and compound words into simpler constituent words, making them easier to process. CRFs are chosen for their probabilistic framework and ability to handle sequential data.
  3. Heuristic Sandhi Corrector: After segmentation, a heuristic Sandhi corrector is applied. This module performs necessary spelling changes to the segmented words, normalizing them into canonical and meaningful simpler words. This involves tasks like insertion, deletion, and substitution to correct inconsistencies and align with standard literary Tamil.

Datasets

The authors collected a corpus of dialectal and spoken Tamil from various online resources, including personal blogs, social media chats, and discussion forums. This corpus was specifically curated to represent different Tamil dialects and their spoken variants. The paper does not mention a pre-existing benchmark dataset for this specific task, implying their collected data serves as a novel resource for this research.

Key Insights and Ideas

  • The "informal internet" is a rich source of linguistic data: The paper underscores how user-generated content on the web is a goldmine for studying real-world language use, including dialectal variations.
  • FSTs are powerful for modeling linguistic regularity: The success of the transformation engine demonstrates the effectiveness of FSTs in capturing and transforming systematic linguistic patterns, particularly suffixations.
  • Agglutination and compounding are key challenges: The need for a dedicated word boundary identifier highlights the morphological complexity of Tamil and the necessity of addressing it for accurate processing.
  • Standardization unlocks downstream NLP performance: The indirect evaluation shows that transforming dialectal text to standard written Tamil significantly improves the quality of machine translation, suggesting this preprocessing step is vital for many NLP applications.

Key Results

  • Direct Evaluation: The DSWT system achieved an encouraging accuracy of over 85% in transforming dialectal spoken Tamil to standard written Tamil. The highest accuracy was observed for the Kongu Tamil dialect (89.7% F-measure), while Madurai and Tirunelveli Tamil dialects showed slightly lower but still strong performance (85.3% and 85.9% F-measure, respectively).
  • Indirect Evaluation: The system demonstrated a significant improvement in the quality of Tamil-English machine translation. When dialectal sentences were first transformed by DSWT and then translated, the quality improved by approximately 40% compared to direct translation of dialectal sentences. Subjective evaluations by native speakers confirmed this improvement, with scores increasing substantially after the dialectal transformation.

Limitations and Future Work

  • One-to-Many Mapping Ambiguity: The paper acknowledges that in cases of one-to-many mapping (where a single spoken form can correspond to multiple written forms), determining the correct written form requires sentence-level context, which the current system does not fully leverage. This is identified as a key area for future work.
  • Contextual Understanding: The current word-level transformation does not account for broader sentence context, which can be crucial for resolving ambiguities in dialectal language.
  • Corpus Availability: The authors note the scarcity of readily available corpora for dialectal Tamil and highlight their own collection as a valuable resource for future research.

Future work will focus on developing methods to resolve the ambiguities arising from one-to-many mappings by incorporating sentence-level context, thereby further enhancing the accuracy and utility of the DSWT system.

Summary updated

BibTeX
@inproceedings{kalimuthu2014automatic,
  author = {Marimuthu Kalimuthu and Sobha Lalitha Devi},
  title = {{Automatic Conversion of Dialectal Tamil Text to Standard Written Tamil Text using FSTs}},
  booktitle = {SIGMORPHON/SIGFSM},
  year = {2014},
  doi = {10.3115/v1/w14-2805},
}
Figure