Building Domain-Specific Taggers without Annotated (Domain) Data

John Miller, Manabu Torii, K. Vijay-ShankerEMNLP 2007

TL;DR

This paper presents a method for building accurate Part-of-Speech (POS) taggers for new domains without requiring any manually annotated domain-specific data. By leveraging existing resources like the Wall Street Journal (WSJ) corpus and raw domain text, they achieve competitive performance in the challenging Biology domain, demonstrating the effectiveness of unsupervised learning for domain adaptation.

What Problem Does This Paper Address?

Part-of-Speech (POS) tagging is a crucial step in many Natural Language Processing (NLP) tasks. However, POS taggers trained on general-purpose text (like the WSJ corpus) often see a significant drop in accuracy when applied to specialized domains (e.g., Biology, Medicine). The primary reason for this degradation is the mismatch in vocabulary and linguistic patterns. Traditionally, the solution is to create a domain-specific annotated corpus, which is expensive, time-consuming, and requires significant funding and resources. This paper tackles the problem of building effective domain-specific taggers without this costly annotation step, aiming to make NLP more accessible to specialized fields.

Methods

The core of the proposed method lies in building a domain-specific lexicon and using an unsupervised learning approach, specifically Expectation-Maximization (EM) training for a Hidden Markov Model (HMM).

Here's a breakdown:

  • Domain Lexicon Creation (Text-Lex):
    • Suffix-based Word Association: The method identifies related words by analyzing suffixes. For instance, if "creation" is in the domain text, the presence of words like "create," "created," and "creations" helps infer that "creation" is likely a noun. This goes beyond simple suffix matching by considering the stem of the word.
  • Feature Vectors: For each word, two types of feature vectors are created:
  • Bin (Binary Count): Indicates the presence or absence of specific suffix-related features.
  • RFreq (Relative Frequency): Captures the relative frequency of these features.
  • kNN for Exemplars: To assign initial POS tags to domain words, a k-Nearest Neighbors (kNN) approach is used. It finds similar words in the WSJ corpus based on their Bin and RFreq feature vectors. The average lexical probabilities from these WSJ exemplars are used to initialize the domain lexicon.
  • Smoothing: A square root smoothing method is applied to the initial lexical probabilities, particularly for low-frequency words, to prevent overfitting to sparse data.
  • Coded Words: Words that don't appear frequently enough to form robust features (e.g., less than 5 times) are categorized as "coded words." These are further classified based on suffixes or as "unknown." Their tags are assigned based on similar aggregated words in the WSJ corpus.
  • HMM Training (Text-EM):
    • Initial Probabilities: The HMM tagger is initialized with:
  • Transitional Probabilities: Smoothed bigram probabilities from the WSJ corpus. This captures general syntactic preferences.
  • Lexical (Emit) Probabilities: Derived from the Text-Lex created in the previous step, based on suffix and orthographic information.
  • EM Algorithm: The EM algorithm is then used to refine these probabilities using raw, unannotated domain text (Text-EM). This unsupervised training adjusts the model to the specific domain's linguistic patterns.
  • Damping Factor: A damping factor is introduced in the EM algorithm to moderate changes in transitional and emit probabilities during iterations. This helps prevent the unsupervised training from spiraling in the wrong direction due to significant differences between the new domain and the WSJ corpus.

Datasets

  • Wall Street Journal (WSJ) Corpus: Used as a source of general-purpose POS-annotated text. It serves as the basis for initial transitional probabilities and as a source of exemplars for building the domain lexicon.
  • Biology Domain Text (Text-Lex and Text-EM):
    • Text-Lex: A subset (about 1%) of Medline abstracts (133,666 titles/abstracts) was used to build the domain-specific lexicon. This text covers broad biomedical topics.
  • Text-EM: The GENIA corpus (version 3.02), specifically the raw text portion (around 2000 abstracts), was used for EM training. This text is from a subfield of molecular biology. The GENIA corpus itself is also POS-annotated, but this annotation is disregarded during the EM training phase to simulate an unsupervised scenario.

Key Insights and Ideas

  • Suffixes as Powerful Indicators: Even in a relatively uninflected language like English, suffix information, when combined with stem analysis and related word identification, provides a strong signal for POS tagging.
  • Leveraging Existing Resources: The paper demonstrates that valuable domain-specific taggers can be built by cleverly combining readily available resources (general corpus, raw domain text) without the need for expensive manual annotation.
  • EM for Domain Adaptation: EM training is a powerful tool for adapting general models to new domains by learning from unannotated domain data, effectively "unlearning" biases from the original training data.
  • The Importance of Lexicon Initialization: The quality of the initial domain lexicon, built using suffix analysis and WSJ exemplars, significantly impacts the effectiveness of the subsequent EM training.
  • Damping for Stability: Introducing a damping factor in the EM algorithm is crucial for stabilizing the learning process and preventing the model from making drastic, potentially incorrect, adjustments early on.

Key Results

  • Competitive Accuracy: The proposed HMM tagger achieved an accuracy of 95.77% on the Biology domain (molecular biology subfield using GENIA corpus).
  • Comparison to State-of-the-Art: This performance is competitive with, and in some cases slightly better than, supervised taggers developed for the Biology domain:
  • MedPost tagger: 94.1%
    • PennBioIE tagger: 95.1%
  • WSJ Tagger Degradation: A tagger trained solely on the WSJ corpus showed significant performance degradation when applied to the Biology domain, highlighting the problem this paper addresses.
  • Lexicon Recall: The developed lexicon achieved a word-token recall of 99.0%.
  • kNN Recall: The kNN method for identifying tags achieved a word-type recall of 96.3%.

Limitations and Future Work

  • Damping Factor Tuning: While the damping factor helps, the authors believe there is scope for further improvement to minimize drift during EM training.
  • kNN Improvement: The kNN method for identifying nearest neighbors could potentially be improved.
  • Automatic Suffix Mining: The authors suggest exploring methods to automatically mine domain-specific suffixes, rather than relying on pre-defined lists.
  • Prefix Information: Prefix information, which was not systematically incorporated, could also be beneficial.
  • More Sophisticated Models: While the paper uses a first-order HMM, future work could explore more complex models that consider more features (e.g., words to the right).
  • Broader Domain Adaptation: The method could be tested on other specialized domains, such as the medical domain, which has a distinct vocabulary from biology.

Summary updated

BibTeX
@inproceedings{miller2007building,
  author = {John Miller and Manabu Torii and K. Vijay-Shanker},
  title = {{Building Domain-Specific Taggers without Annotated (Domain) Data}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2007},
}
Figure