A Beam-Search Decoder for Normalization of Social Media Text with Application to Machine Translation

Pidong Wang, Hwee Tou NgNAACL 2013 · OpenReview

Abstract: Pidong Wang, Hwee Tou Ng. Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

2013.

Introduction

Natural Language Processing (NLP) systems, particularly Machine Translation (MT), are typically optimized for formal text such as news reports or legal documents. However, the rise of social media platforms like Twitter and Weibo has introduced a significant challenge: "noisy" text. Informal communication often features non-standard abbreviations, omitted punctuation, and grammatical shortcuts. These deviations cause a sharp decline in performance for downstream tasks; for instance, part-of-speech tagging accuracy can drop from 90% on news text to 76% on social media content.

While previous research has largely focused on word-level substitutions (e.g., replacing "u" with "you"), this paper argues that effective normalization requires a holistic approach that addresses structural issues like missing pronouns and punctuation. The authors propose a beam-search decoder framework designed to transform informal social media posts into formal language suitable for standard MT systems.

The Beam-Search Framework

Rather than relying on a simple rule-based pipeline, the proposed system treats normalization as a search problem. The goal is to find the most probable formal sequence given an informal input string.

Example of the normalization search process for the input 'whr u'.

The framework utilizes several key components:

  • Hypothesis Producers: Modular engines that suggest potential transformations, including dictionary lookups, pronunciation-based similarity (Pinyin for Chinese, phone-mapping for English), and rules for abbreviations.
  • Structural Correction: A two-layer Dynamic Conditional Random Field (DCRF) model predicts punctuation and sentence boundaries, while a separate CRF model identifies and restores missing words (e.g., "be" verbs in English or dropped pronouns in Chinese).
  • Scoring and Tuning: A linear model evaluates hypotheses by combining language model scores with various penalty and reward features. The weights for these features are optimized using Pairwise Ranking Optimization (PRO).

To ensure linguistic coherence, the decoder employs a contextual filter that checks proposed formal words against a 4-gram language model derived from a large formal corpus.

Data and Evaluation

The study introduced two new manually annotated corpora to facilitate research in this area:

  1. Chinese-English (CN2EN): 1,000 Weibo messages normalized and translated into formal English.
  2. English-Chinese (EN2CN): 2,000 SMS messages from the NUS SMS corpus, normalized and translated into formal Chinese.

These datasets allow for the evaluation of both normalization quality and the subsequent impact on machine translation performance.

Results and Discussion

The experimental results demonstrate that the beam-search approach significantly outperforms standard baselines, including phrase-based machine translation (PBMT) approaches to normalization.

BLEU score improvements for normalization and machine translation across different system configurations.

Key findings include:

  • Normalization Gains: The decoder achieved a BLEU score improvement of 9.98% for Chinese and 7.35% for English over baseline methods.
  • MT Performance: Feeding the normalized text into a standard MT system resulted in a quality boost of 1.38 BLEU points for Chinese-to-English and 1.35 BLEU points for English-to-Chinese translation.
  • Structural Importance: The authors found that fixing punctuation and restoring missing words are not merely cosmetic; they provide essential structural cues that allow MT systems to correctly identify sentence boundaries and clause structures.

Furthermore, the system is highly efficient, processing messages at an average rate of 0.2 seconds per message, making it viable for real-time applications.

Conclusion

This research highlights normalization as a "force multiplier" for NLP. By investing in a high-quality normalization pre-processor, researchers can leverage existing, powerful MT systems trained on formal text without the need for massive, scarce social media training data. Future work may explore tighter integration between normalization and translation, such as passing normalization lattices to the MT engine to mitigate error propagation in the pipeline.

Summary updated

BibTeX
@inproceedings{wang2013beam,
  author = {Pidong Wang and Hwee Tou Ng},
  title = {{A Beam-Search Decoder for Normalization of Social Media Text with Application to Machine Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2013},
  url = {https://openreview.net/forum?id=HmB3tawAYqX},
}
Figure