Joint POS Tagging and Text Normalization for Informal Text

Chen Li, Yang LiuIJCAI 2015

TL;DR

This paper tackles the challenge of accurately processing informal text (like tweets and SMS) by proposing a novel joint approach to simultaneously normalize non-standard words and assign part-of-speech (POS) tags. By integrating these two tasks within a single Viterbi decoding framework, the authors demonstrate significant improvements in both normalization and POS tagging accuracy compared to pipeline methods.

What Problem Does This Paper Address?

Informal text, prevalent in social media, is rife with challenges for traditional Natural Language Processing (NLP) tools. This includes misspellings, abbreviations, and other non-standard word forms. These irregularities make it difficult for systems to accurately identify the grammatical role (POS tagging) and meaning of words. Previous work has tackled text normalization and POS tagging separately. However, this paper argues that these tasks are interdependent: accurate normalization can improve POS tagging, and POS information can aid normalization. The core problem is the lack of a unified approach that leverages this interdependence to improve performance on both tasks simultaneously for informal text.

Methods

The paper introduces a joint Viterbi decoding process that simultaneously determines a token's POS tag and its normalized form. This is a departure from pipeline approaches where normalization is performed first, followed by POS tagging on the normalized text.

The key innovation lies in how the Viterbi decoding trellis is structured. For each token in the input sequence, a state in the trellis now represents a combination of a POS tag and a normalization word candidate. Crucially, these states also incorporate the context of the previous and next words (specifically, their normalization candidates) to better capture dependencies.

The forward score calculation for each state combines several probabilities:

  • Emission probabilities: The likelihood of a POS tag given the normalized word and its context (from the POS tagger) and the likelihood of a normalization candidate for the token (from the normalization model).
  • Transition probabilities: The probability of transitioning between POS tags and the probability of a sequence of normalized words based on a trigram language model.

The authors use a weighted combination of these probabilities, with parameters $\alpha$ and $\beta$ to balance the contributions.

Datasets

The authors created two new datasets for this joint task:

  • Test Data Set 1: Derived from [Owoputi et al., 2013]'s OCT27 and DAILY547 datasets. This set contains 798 tweets that were manually annotated by native English speakers to identify non-standard tokens and their correct forms, in addition to existing POS tag labels.
  • Test Data Set 2: Derived from [Han and Baldwin, 2011]'s dataset of 549 tweets, which already contained non-standard tokens and their corrections. POS tags were added to this dataset by training a POS tagger on the Owoputi data and then manually correcting the tags.

Both datasets are characterized by having at least one non-standard token per sentence.

Key Insights and Ideas

  • Interdependence of Normalization and POS Tagging: The paper strongly demonstrates that non-standard words negatively impact POS tagging accuracy, not just for themselves but also for surrounding words due to context sensitivity. Conversely, normalizing these words can significantly improve POS tagging.
  • Benefit of Joint Decoding: A joint approach outperforms pipeline methods by avoiding the propagation of errors from an imperfect normalization step to the POS tagging step. It allows POS information to inform normalization and vice-versa.
  • Context is Crucial: Incorporating contextual words (previous and next tokens) into the state representation of the Viterbi trellis is vital for both accurate POS tagging and normalization, especially in informal text.
  • Trade-off Between Speed and Accuracy: While the joint model is computationally more expensive, the authors show that using a limited number of normalization candidates (e.g., 5) can achieve performance close to using many more candidates, offering a practical balance between speed and accuracy.

Key Results

  • Improved POS Tagging: The joint decoding system consistently outperforms pipeline methods and systems without normalization. For instance, on Test Set 1, the joint decoding system achieves 91.21% POS tagging accuracy, compared to 90.8% for a pipeline approach with first-order normalization and 90.91% with second-order normalization.
  • Improved Normalization: The joint decoding approach also shows improvements in normalization accuracy, outperforming pipeline methods. On Test Set 1, joint decoding achieved 77.31% normalization accuracy, compared to 76.96% for the pipeline with second-order normalization.
  • Oracle Performance: The "Oracle Normalization + POS tagging" setup, where non-standard words are perfectly normalized, yields the highest POS tagging accuracy (92.05% on Test Set 1), highlighting the significant potential for improvement through better normalization.
  • Impact on Standard Words: Normalization has a much smaller impact on the POS tagging of standard words compared to non-standard words, as shown by the error analysis.

Limitations and Future Work

  • Computational Cost: The joint decoding approach is computationally more expensive than pipeline methods, although the authors suggest practical ways to mitigate this (e.g., limiting candidate numbers).
  • Data Annotation: The paper highlights the scarcity of fully annotated data (both POS tags and normalization) for training. The created datasets are primarily for evaluation.
  • Future Work: The authors propose creating a training set with both POS and normalization annotations to enable joint training strategies, which could further improve performance. They also suggest exploring more sophisticated joint training methods.

Summary updated

BibTeX
@inproceedings{li2015joint,
  author = {Chen Li and Yang Liu},
  title = {{Joint POS Tagging and Text Normalization for Informal Text}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2015},
}
Figure