Normalization of Text Messages Using Character- and Phone-based Machine Translation Approaches

Chen Li, Yang Liu

Introduction

The proliferation of social media and SMS has introduced a significant challenge for Natural Language Processing (NLP) and Text-to-Speech (TTS) systems: non-standard text. Users frequently employ abbreviations, phonetic spellings, and "leet-speak" (e.g., "2mr" for "tomorrow"), creating a massive "out-of-vocabulary" (OOV) problem. Traditional systems trained on clean data like news articles struggle to interpret these informal dialects. This research proposes a robust normalization framework that treats the task as a translation problem, converting informal "txt spk" into standard English.

Methodology

The core of the proposed approach is a hybrid ensemble using the Moses statistical machine translation toolkit. The system integrates two distinct strategies to handle the diverse ways users abbreviate words.

Single-step Character-based MT (System A)

This model treats normalization as a direct character-to-character translation. It learns transformation patterns, such as mapping "@" to "at" or "u" to "you," based on surface-level visual similarities.

Two-stage Phonetic MT (Systems B & C)

The primary innovation is a phonetic "bridge" model. Many abbreviations are phonetically motivated rather than visually derived.

  1. Stage 1 (System B): Translates the non-standard character sequence into a phonetic sequence using International Phonetic Alphabet (IPA) symbols.
  2. Stage 2 (System C): Translates the IPA sequence into a standard English word.

By "sounding out" the word, the system can identify matches like "snds" and "sounds" which share few characters but are phonetically nearly identical.

System Combination

The final output is generated by merging the N-best lists from both the character-based and phonetic-based models. A heuristic approach prioritizes the single-step model when confidence is high but leverages the phonetic model for more complex cases.

Flowchart of the hybrid normalization system combining character-based and phonetic-based MT.

Experimental Setup

The system was evaluated using a benchmark of 303 isolated non-standard words from SMS messages. Training data included 6,532 pairs of non-standard/standard words from Twitter and an offline dictionary of 75,262 English words with IPA pronunciations. A 5-gram character language model and a phone-based language model were used to guide the translation process.

Results and Discussion

The hybrid system achieved a Top-1 accuracy of 71.96%, significantly outperforming previous state-of-the-art methods and simple spell-checkers.

Accuracy comparison of individual components and the combined system across different N-best list sizes.

As shown in the results, the two-stage phonetic model, while having lower Top-1 accuracy than the single-step model, provides high "coverage." The correct word is often present within its Top-10 or Top-20 candidates. The combination effectively exploits these complementary strengths, allowing the system to correctly normalize words like "btiful" (beautiful) and "lrg" (large) where surface-level models fail.

Conclusion

By treating text normalization as a machine translation task and incorporating phonetic information, this research provides a powerful tool for cleaning social media data. Future work will focus on incorporating sentence-level context to further improve disambiguation (e.g., distinguishing "2" as "to," "too," or "two") and evaluating the impact on downstream TTS performance.

Summary updated

BibTeX
@inproceedings{li2012normalization,
  author = {Chen Li and Yang Liu},
  title = {{Normalization of Text Messages Using Character- and Phone-based Machine Translation Approaches}},
  booktitle = {Interspeech},
  year = {2012},
  doi = {10.21437/interspeech.2012-611},
}
Figure