Towards Robust Cross-Domain Domain Adaptation for Part-of-Speech Tagging

Tobias Schnabel, Hinrich SchützeIJCNLP 2013 · OpenReview

Tobias Schnabel and Hinrich Schütze (2013) address the inherent brittleness of Part-of-Speech (POS) taggers when applied outside their training domain. While a model trained on financial news (e.g., the Wall Street Journal) may perform well on similar text, its accuracy often plummets when encountering the "unknown words" prevalent in weblogs, emails, or specialized biomedical literature. This paper investigates whether a universal set of features or algorithms exists for unsupervised domain adaptation (DA) across diverse target domains.

Methodology: SVM-Based Word Classification

Departing from complex sequence labeling models like Conditional Royal Fields (CRFs), the authors utilize a simpler word classification approach using Support Vector Machines (SVMs). This choice facilitates large-scale experimentation and proves surprisingly robust for tagging unknown words. The feature set incorporates:

  • Distributional Features: The local context (immediate neighbors) of a word.
  • Shape Features: Morphological cues such as suffixes, capitalization, and digit patterns.

A key innovation introduced is Training Set Filtering. Instead of using the entire source domain for training, the authors restrict the training set to "long" words (length $|w| > k$). The hypothesis is that longer words are more representative of the productive linguistic processes—like complex suffixation—that characterize unknown words in a new domain.

The Similarity Paradox and Feature Robustness

The study reveals that the effectiveness of DA techniques is highly dependent on the similarity between the source and target domains. This "Similarity Paradox" means that methods beneficial for domains close to the source (e.g., weblogs) can be detrimental for highly dissimilar domains (e.g., biomedical text).

For instance, while shape features (suffixes) are generally helpful for web-based text, they can decrease performance in the biomedical domain. Similarly, Dimensionality Reduction via Singular Value Decomposition (SVD), often considered a standard tool in DA, was found to help significantly only in the biomedical domain, while consistently hurting performance across the five web domains tested.

Performance comparison across domains using different filtering and normalization techniques.

As shown in Figure 2, the performance of various configurations varies wildly across domains. Notably, simple distributional features (context) proved to be the most robust across all tested environments, whereas shape-based features were far more volatile.

Key Results: Breakthrough in Biomedical Tagging

The most striking result of the paper is the performance gain on the Penn BioTreebank. By combining training set filtering with the omission of shape features, the authors achieved an accuracy of 88.13% for unknown words. This represents a 10% absolute improvement over previous state-of-the-art results (76.3%).

Impact of training set filtering on unknown word accuracy in the biomedical domain.

Figure 3 illustrates the impact of the filtering threshold $k$. For the biomedical domain, a threshold of $|w| > 7$ provided a significant boost, aligning the training distribution more closely with the characteristics of unknown words in the target domain.

In the web domains (SANCL 2012 shared task), the authors established that simple word classification could match or even exceed the performance of more complex sequence models for unknown word prediction. They also noted that tf (term frequency) weighting for context words generally outperformed binary weighting, suggesting that frequency provides a valuable signal for POS tagging.

Conclusion and Future Directions

The research underscores that there is no "one-size-fits-all" solution for domain adaptation in POS tagging. The choice of features, dimensionality reduction, and training data filtering must be tuned to the specific target domain. Future work involves developing predictive diagnostics—perhaps based on Jensen-Shannon divergence—to determine the optimal DA strategy for a new domain without exhaustive empirical testing. Additionally, integrating these filtered classification techniques into full sequence-labeling frameworks could further enhance overall tagging accuracy.

Tobias Schnabel, Hinrich Schütze. Proceedings of the Sixth International Joint Conference on Natural Language Processing. 2013.

Summary updated

BibTeX
@inproceedings{schnabel2013towards,
  author = {Tobias Schnabel and Hinrich Schütze},
  title = {{Towards Robust Cross-Domain Domain Adaptation for Part-of-Speech Tagging}},
  booktitle = {IJCNLP},
  year = {2013},
  url = {https://openreview.net/forum?id=B-7xU2XbbKu},
}
Figure