TL;DR
This paper introduces a generative Bayesian model for part-of-speech (POS) tagging that treats tags like "topics" in a Latent Dirichlet Allocation (LDA) framework. By enforcing sparse priors—reflecting the reality that most words have one dominant tag—and explicitly modeling "ambiguity classes" based on word morphology, the authors achieved a massive 57% error reduction over previous state-of-the-art semi-supervised methods.
What Problem Does This Paper Address?
While supervised POS taggers are incredibly accurate (97%+) on familiar data, they stumble when encountering unknown words or shifting to new domains (dropping to 70% or lower). The "gold standard" requires massive amounts of human-labeled data, which simply doesn't exist for most of the world's languages.
The authors tackle the challenge of semi-supervised tagging, where the model only has access to a "tagging dictionary" (a list of possible tags for some words) and a large pile of unlabeled text. The goal is to bridge the gap between fully unsupervised learning (which is very hard) and fully supervised learning (which is very expensive).
Methods
The authors depart from the traditional Hidden Markov Model (HMM) approach, which focuses on sequences of tags. Instead, they build on Latent Dirichlet Allocation (LDA).
- The LDA Extension: In standard LDA, documents are mixtures of topics. Here, a "document" is the collection of all contexts in which a specific word type appears. The "topics" are the POS tags.
- Sparse Priors: They incorporate a Dirichlet prior ($\alpha < 1$) on the distribution of tags for each word. This encodes the linguistic intuition that while a word like "walk" can be a noun or a verb, it usually functions as one much more frequently than the other.
- Ambiguity Class Model: This is a key innovation. The model doesn't just guess tags; it models the set of possible tags a word can take. It uses morphological features (like the suffix "-s" or capitalization) to predict which ambiguity class a word belongs to.
- Context Features: Instead of just looking at the previous tag (like an HMM), the model generates a four-word window (two words before, two words after) as observed features to predict the hidden tag.
Datasets
The model was evaluated using the Wall Street Journal (WSJ) Penn Treebank.
- Tags: They used a coarse-grained set of 17 tags (simplified from the original 45).
- Dictionary Settings: To test robustness, they used three levels of supervision:
- A complete dictionary of all words in the corpus.
- A cutoff of 2, including only words appearing at least twice in the test set.
- A cutoff of 3, including only words appearing at least three times.
- Unlabeled Data: 49,208 sentences from the WSJ.
Key Insights and Ideas
- Sparsity is a Feature, Not a Bug: Previous models often struggled because they allowed for too much ambiguity. By forcing the model to favor sparse distributions (one dominant tag per word), the authors aligned the math with how language actually works.
- Morphology as a "Cold Start" Solution: For words not in the dictionary, the model uses suffixes and casing to narrow down the possibilities. For example, a word ending in "-ing" is highly likely to belong to an ambiguity class containing "Verb."
- Context over Transitions: The results suggest that modeling the local lexical context (the actual words nearby) is often more informative for semi-supervised learning than modeling the sequence of hidden tags (the transition probabilities).
Key Results
The full model (LDA + AC, for Ambiguity Class) consistently outperformed all baselines and previous state-of-the-art models.
- Accuracy: On the complete dictionary task, it reached 93.4% accuracy.
- Superiority over HMMs: It significantly beat both Maximum Likelihood HMMs (83.2%) and Bayesian HMMs (87.3%).
- Impact of Ambiguity Modeling: When the dictionary was sparse (cutoff $\ge$ 3), adding the Ambiguity Class model improved accuracy from 85.0% to 89.7%.
- State-of-the-Art: Compared to the previous best model (Contrastive Estimation + spelling), this model reduced the error rate by up to 57% in low-supervision settings.
Limitations and Future Work
- Model "Deficiency": The authors admit the model is "deficient" because it generates the same word token multiple times (once for each context it appears in). While theoretically imperfect, this didn't seem to hurt performance.
- Prototype Scaling: The current model relies on a dictionary. Future work could explore using a very small set of "prototype" words for each tag instead of a full dictionary.
- Task Expansion: The authors suggest this Bayesian LDA-style approach could be applied to other NLP tasks like Named Entity Recognition (NER) or inducing deeper syntactic structures.
Summary updated