Mining for unambiguous instances to adapt part-of-speech taggers to new domains

Dirk Hovy, Barbara Plank, Héctor Martínez Alonso, Anders Søgaard

Introduction

Adapting Part-of-Speech (POS) taggers to "noisy" domains like Twitter usually requires expensive manual labeling. This research introduces a clever, low-cost alternative: mining massive unlabeled datasets for sentences where every word is "unambiguous" according to a dictionary (like Wiktionary). By adding these "free" labeled examples to standard training data, significant accuracy boosts are achieved across multiple languages and domains.

The Problem: Domain Mismatch

Most POS taggers are trained on "Newswire" data, such as the Wall Street Journal, which is grammatically pristine. When these models are applied to Twitter, spoken transcripts, or search queries, performance plummets. This is due to domain mismatch: the vocabulary, slang, and syntax of social media or spoken language are alien to a model expecting formal prose. Traditional manual annotation is slow and expensive; this approach instead leverages the billions of unlabeled tweets and queries already available.

Methodology

The proposed "parameter-free" approach to domain adaptation involves three key steps:

  1. Dictionary Mining: Using Wiktionary to identify words that are unambiguous (i.e., they only have one possible POS tag). In English, roughly 93% of unique word types in Wiktionary fall into this category.
  2. Instance Extraction: Scanning massive unlabeled corpora from the target domain. If a sentence consists entirely of unambiguous words, it is treated as a labeled training instance. For example, the search query "peter pan williams movie" becomes a high-confidence training example because all its components are unambiguous in the dictionary.
  3. Cluster Expansion: To handle the "noise" of social media (like typos or elongated words), the authors use Brown Clustering. This groups words that appear in similar contexts. If a known unambiguous word is in a cluster with unknown words, the tag is projected onto the unknown words, effectively expanding the dictionary.
Example of cluster-based tag projection for noisy tokens.

Datasets and Evaluation

The models were trained using standard newswire corpora and augmented with mined data from 57 million English tweets, as well as millions of tweets in Spanish, Portuguese, and Dutch. The authors also utilized the Switchboard corpus for spoken language and 10 million search queries. Evaluation was conducted on existing Twitter benchmarks and new manually-annotated test sets for Spanish and Portuguese.

Key Results and Insights

The approach, labeled CRF+D' (CRF + mined data + clusters), consistently outperformed off-the-shelf tools and standard models.

POS tagging accuracy across domains and languages.
  • Accuracy Improvements: English Twitter accuracy rose from 80.55% to 87.76%, while Spanish Twitter accuracy increased from 75.66% to 82.87%.
  • Reducing OOV Rates: The primary driver of success is the drastic reduction in the "Out-of-Vocabulary" (OOV) rate. By seeing domain-specific words in the mined data, the model no longer has to guess their tags blindly.
  • Short Text Advantage: This method is uniquely suited for "short-form" domains. The shorter the text, the higher the probability that every word is unambiguous, making it perfect for tweets and queries.

Limitations and Future Work

The "unambiguous" requirement is a harsh filter; in domains with long, complex sentences, the amount of mineable data drops significantly. Additionally, the mined data is heavily skewed toward nouns, which helps vocabulary coverage but may not assist in learning complex new syntactic structures. Future work may explore "softer" ways to include nearly-unambiguous sentences without introducing excessive noise.

Summary updated

Abstract

We present a simple, yet effective approach to adapt part-of-speech (POS) taggers to new domains. Our approach only requires a dictionary and large amounts of unlabeled target data. The idea is to use the dictionary to mine the unlabeled target data for unambiguous word sequences, thus effectively collecting labeled target data. We add the mined instances to available labeled newswire data to train a POS tagger for the target domain. The induced models significantly improve tagging accuracy on held-out test sets across three domains (Twitter, spoken language, and search queries). We also present results for Dutch, Spanish and Portuguese Twitter data, and provide two novel manually-annotated test sets.
BibTeX
@inproceedings{hovy2015mining,
  author = {Dirk Hovy and Barbara Plank and Héctor Martínez Alonso and Anders Søgaard},
  title = {{Mining for unambiguous instances to adapt part-of-speech taggers to new domains}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
  doi = {10.3115/v1/n15-1135},
}
Figure