Zipfian corruptions for robust POS tagging

Anders SøgaardNAACL 2013 · OpenReview

Abstract: Inspired by robust generalization and adversarial learning, we describe a novel approach to learning structured perceptrons for part-of-speech (POS) tagging that is less sensitive to domain shifts. The objective of our method is to minimize average loss under random distribution shifts. We restrict the possible target distributions to mixtures of the source distribution and random Zipfian distributions. Our algorithm is used for POS tagging and evaluated on the English Web Treebank and the Danish Dependency Treebank with an average 4.4% error reduction in tagging accuracy.

Introduction

Supervised machine learning models are notoriously brittle when faced with domain shift. A POS tagger trained on the Wall Street Journal (WSJ) might achieve 97% accuracy on similar news text but drop to 89% when applied to the "Wild West" of Web 2.0 content like emails, blogs, and forums.

The primary culprit is feature swamping. In standard training, a model learns to rely heavily on highly indicative features (e.g., specific frequent words). However, in a new domain, these features might be absent (Out-of-Vocabulary or OOV effects), leaving the model to rely on "swamped" features that were not properly weighted during training. While semi-supervised or transfer learning can mitigate this, they often require large unlabeled corpora from the target domain, which are not always available.

Methodology: Simulating the Unknown

The proposed approach, dubbed $Z_3SP$, utilizes adversarial learning and Zipf’s Law to build robustness without target-domain data. The core idea is that if we assume a target domain will be "Zipfian-different," we can simulate those differences during the training phase.

Figure 1: Illustration of how different fits (2d vs 1d) handle missing features, demonstrating the intuition behind robust decision boundaries.

Instead of training on "clean" source data, the algorithm introduces "soft" corruptions. For every training instance, the model samples a vector from a Zipfian distribution to reweight the feature vector. Unlike previous "feature bagging" methods that randomly delete features (binary 0/1), this method scales them. By forcing the model to minimize loss under these simulated shifts, it learns to prioritize features that remain robust across various Zipfian versions of the data, preventing the model from putting "all its eggs in one basket."

Experimental Setup

The model was trained on "clean" gold-standard data and evaluated on diverse, "messy" datasets:

  • English: Trained on Sections 2–21 of the Wall Street Journal (WSJ). Evaluated on the English Web Treebank (EWT), covering answers, newsgroups, reviews, weblogs, and emails.
  • Danish: Trained on the newspaper section of the Danish Dependency Treebank (DDT). Evaluated on sections covering law, literature, and magazines.

Key Results

The $Z_3SP$ model consistently outperformed both the standard Structured Perceptron (SP) and a Binary Subspace Perceptron (BSP) baseline, achieving a 4.4% average error reduction across all test sets.

Figure 2: Comparison of tagging accuracy across English (EWT) and Danish (DDT) sub-domains, showing the consistent performance gains of the Z3SP model.

As shown in the results table, significant gains were observed in "messy" domains like newsgroups (87.42% vs 86.82% SP) and specialized domains like Law (93.35% vs 92.38% SP). These improvements were statistically significant ($p < 0.01$) across almost all domains, validating the Zipfian assumption as a powerful heuristic for linguistic data.

Conclusion and Future Work

This method is elegantly simple, requiring only a minor modification to the weight update step of a standard perceptron. Because the corruption occurs during training, the model remains as fast at test time as a standard implementation.

While highly effective for POS tagging, future work could explore the sensitivity of the Zipfian parameter across different languages and its applicability to other structured prediction tasks, such as dependency parsing. Additionally, while the Zipfian mixture assumption holds for many natural language tasks, its performance in highly technical domains with non-standard power laws remains an open question.

Summary updated

Abstract

Inspired by robust generalization and adversarial learning we describe a novel approach to learning structured perceptrons for part-ofspeech (POS) tagging that is less sensitive to domain shifts. The objective of our method is to minimize average loss under random distribution shifts. We restrict the possible target distributions to mixtures of the source distribution and random Zipfian distributions. Our algorithm is used for POS tagging and evaluated on the English Web Treebank and the Danish Dependency Treebank with an average 4.4% error reduction in tagging accuracy.
BibTeX
@inproceedings{sgaard2013zipfian,
  author = {Anders Søgaard},
  title = {{Zipfian corruptions for robust POS tagging}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2013},
  url = {https://openreview.net/forum?id=Bk-9FXZd-r},
}
Figure