Normalization of Transliterated Words in Code-Mixed Data Using Seq2Seq Model & Levenshtein Distance

Soumil Mandal, Karthick NanmaranEMNLP 2018

TL;DR

This paper tackles the challenge of normalizing phonetically transliterated words in code-mixed social media text, a common issue in multilingual communities. They propose a novel two-stage approach combining a character-level seq2seq model for initial normalization with Levenshtein distance for matching against a standard transliteration lexicon, achieving a high accuracy of 90.27%.

What Problem Does This Paper Address?

Social media platforms are increasingly used for communication across different languages, leading to "code-mixing" where users seamlessly switch between languages within a single conversation. A significant challenge arises when users type words from one language (e.g., Bengali) using the script of another (e.g., English Roman script) via phonetic transliteration. This often results in inconsistent spellings and variations due to individual phonetic judgments, dialects, or informal typing habits. For instance, the same Bengali word might be transliterated in multiple ways. This inconsistency hinders the effectiveness of downstream NLP tasks like sentiment analysis, summarization, and machine translation, which rely on standardized text. This paper aims to bridge this gap by developing a robust method to normalize these phonetically transliterated words into their standard forms.

Methods

The authors propose a two-stage modular approach:

  1. First Normalization Module (Seq2Seq Model): This module uses a character-level sequence-to-sequence (seq2seq) model, specifically a stacked LSTM-based architecture. The model is trained to map user-typed phonetic transliterations (input) to their standard transliterations (output). This stage aims to perform an initial normalization, bringing the input string closer to its canonical form. The architecture is chosen for its success in complex sequence mapping tasks like machine translation. Notably, attention mechanisms were omitted due to the relatively small size of word vectors.
  2. Second Normalization Module (Levenshtein Distance): This module takes the output from the first module and compares it against a dictionary of standard transliterations (BN_TRANS). It employs the Levenshtein distance (LD) algorithm, which measures the minimum number of single-character edits (insertions, deletions, or substitutions) required to change one word into another. The candidate string with the closest match (minimum LD) is returned as the final normalized form. A modified version of LD was also explored, where certain phonetically similar character pairs (e.g., 'a'/'o', 'b'/'v') were treated as having zero cost to account for common transliteration variations.

Datasets

The paper introduces and utilizes three datasets:

  • Parallel Lexicon (PL): This dataset consists of two columns: phonetically transliterated Bengali words from code-mixed data and their corresponding standard ITRANS transliterations. It was manually created by back-transliterating to Bengali script and then converting to ITRANS. The final size was 6000 entries.
  • Transliteration Dictionary (BN_TRANS): This lexicon contains Bengali words in their native Eastern Nagari script and their standard ITRANS transliterations. It comprises 21,850 entries.
  • Test Data: This data was derived from previously tagged Bengali-English code-mixed data. After manual checking and filtering, it resulted in 905 tagged tokens. These tokens were then tagged with their ITRANS using the same approach as for the PL dataset.

Initial rule-based normalization techniques were applied to the PL and test data, including replacing digits with their phonetic equivalents and trimming consecutive identical characters to a maximum of two.

Key Insights and Ideas

  • The Power of Two Stages: The paper demonstrates that a sequential approach, where an initial seq2seq normalization primes the text before a lexicon-based matching, significantly outperforms direct lexicon matching. This suggests that seq2seq models can effectively capture and correct phonetic variations that simple string matching might miss.
  • Character-Level Seq2Seq for Phonetics: Using a character-level seq2seq model is crucial for handling the fine-grained phonetic variations inherent in transliterated text. This approach avoids the need for pre-defined grapheme-to-phoneme mappings.
  • Levenshtein Distance as a Refinement: While seq2seq provides a good initial normalization, Levenshtein distance offers a robust way to find the closest match within a structured lexicon, ensuring the output adheres to a standardized form.
  • Phonetic Similarity Matters: The exploration of a modified Levenshtein distance that treats phonetically similar characters as equivalent highlights the importance of incorporating linguistic knowledge about pronunciation variations into the normalization process.

Key Results

  • Word-Level Accuracy: The proposed model (setup_4, using seq2seq and modified LD) achieved an accuracy of 90.27% on the test data. This is a significant improvement over setups that did not use seq2seq normalization.
  • Task-Level Improvement: When applied to a sentiment analysis task on Bengali-English code-mixed data, the normalized text led to an improvement in accuracy by 1.5% (from 80.97% to 82.47%). Precision, Recall, and F1 scores also saw similar improvements.
  • Training Data Impact: The authors note that the accuracy achieved during training (around 41.2%) was relatively low, attributing it to the limited amount of training data and the high degree of phonetic variation. However, the performance on the testing data was considerably better.

Limitations and Future Work

  • Data Requirement: The need for a parallel corpus (PL) for training the seq2seq model is acknowledged as a limitation, as manual creation and back-transliteration are tedious.
  • Processing Speed: The current approach, involving both seq2seq and Levenshtein distance calculations, is computationally intensive, leading to a slower processing speed.
  • Out-of-Vocabulary (OOV) Words: A significant portion of errors (92%) was due to standard forms not being present in the BN_TRANS lexicon (OOV words), which included slangs and expressions.
  • Future Directions:
    • Collecting more transliterated words and increasing the size of the PL and BN_TRANS datasets.
  • Improving processing speed by exploring simpler string matching methods or search area reduction algorithms.
  • Adding a lexical checker to identify already normalized words before processing.
  • Integrating the normalization module with context-capturing systems.
  • Expanding the system to handle other Indic languages.
  • Exploring back-transliteration and using Bengali sentiment lexicons for further task-level improvements.

Summary updated

BibTeX
@inproceedings{mandal2018normalization,
  author = {Soumil Mandal and Karthick Nanmaran},
  title = {{Normalization of Transliterated Words in Code-Mixed Data Using Seq2Seq Model \& Levenshtein Distance}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure