Abstract: Inspired by robust generalization and adversarial learning, we describe a novel approach to learning structured perceptrons for part-of-speech (POS) tagging that is less sensitive to domain shifts. The objective of our method is to minimize average loss under random distribution shifts. We restrict the possible target distributions to mixtures of the source distribution and random Zipfian distributions. Our algorithm is used for POS tagging and evaluated on the English Web Treebank and the Danish Dependency Treebank with an average 4.4% error reduction in tagging accuracy.
Introduction
Supervised machine learning models are notoriously brittle when faced with domain shift. A POS tagger trained on the Wall Street Journal (WSJ) might achieve 97% accuracy on similar news text but drop to 89% when applied to the "Wild West" of Web 2.0 content like emails, blogs, and forums.
The primary culprit is feature swamping. In standard training, a model learns to rely heavily on highly indicative features (e.g., specific frequent words). However, in a new domain, these features might be absent (Out-of-Vocabulary or OOV effects), leaving the model to rely on "swamped" features that were not properly weighted during training. While semi-supervised or transfer learning can mitigate this, they often require large unlabeled corpora from the target domain, which are not always available.
Methodology: Simulating the Unknown
The proposed approach, dubbed $Z_3SP$, utilizes adversarial learning and Zipf’s Law to build robustness without target-domain data. The core idea is that if we assume a target domain will be "Zipfian-different," we can simulate those differences during the training phase.
Instead of training on "clean" source data, the algorithm introduces "soft" corruptions. For every training instance, the model samples a vector from a Zipfian distribution to reweight the feature vector. Unlike previous "feature bagging" methods that randomly delete features (binary 0/1), this method scales them. By forcing the model to minimize loss under these simulated shifts, it learns to prioritize features that remain robust across various Zipfian versions of the data, preventing the model from putting "all its eggs in one basket."
Experimental Setup
The model was trained on "clean" gold-standard data and evaluated on diverse, "messy" datasets:
- English: Trained on Sections 2–21 of the Wall Street Journal (WSJ). Evaluated on the English Web Treebank (EWT), covering answers, newsgroups, reviews, weblogs, and emails.
- Danish: Trained on the newspaper section of the Danish Dependency Treebank (DDT). Evaluated on sections covering law, literature, and magazines.
Key Results
The $Z_3SP$ model consistently outperformed both the standard Structured Perceptron (SP) and a Binary Subspace Perceptron (BSP) baseline, achieving a 4.4% average error reduction across all test sets.
As shown in the results table, significant gains were observed in "messy" domains like newsgroups (87.42% vs 86.82% SP) and specialized domains like Law (93.35% vs 92.38% SP). These improvements were statistically significant ($p < 0.01$) across almost all domains, validating the Zipfian assumption as a powerful heuristic for linguistic data.
Conclusion and Future Work
This method is elegantly simple, requiring only a minor modification to the weight update step of a standard perceptron. Because the corruption occurs during training, the model remains as fast at test time as a standard implementation.
While highly effective for POS tagging, future work could explore the sensitivity of the Zipfian parameter across different languages and its applicability to other structured prediction tasks, such as dependency parsing. Additionally, while the Zipfian mixture assumption holds for many natural language tasks, its performance in highly technical domains with non-standard power laws remains an open question.
Summary updated