Multilingual Part-of-Speech Tagging: Two Unsupervised Approaches

Tahira Naseem, Benjamin Snyder, Jacob Eisenstein, Regina Barzilay

Abstract

We demonstrate the effectiveness of multilingual learning for unsupervised part-of-speech tagging. The central assumption of our work is that by combining cues from multiple languages, the structure of each becomes more apparent. We consider two ways of applying this intuition to the problem of unsupervised part-of-speech tagging: a model that directly merges tag structures for a pair of languages into a single sequence and a second model which instead incorporates multilingual context using latent variables. Both approaches are formulated as hierarchical Bayesian models, using Markov Chain Monte Carlo sampling techniques for inference. Our results demonstrate that by incorporating multilingual evidence we can achieve impressive performance gains across a range of scenarios. We also found that performance improves steadily as the number of available languages increases.

Introduction

Part-of-speech (POS) tagging is a foundational task in Natural Language Processing, yet high-performing supervised systems require extensive human-annotated data—a resource unavailable for the vast majority of the world's 7,000+ languages. Unsupervised tagging attempts to learn these patterns from raw text alone but faces a major hurdle: linguistic ambiguity. For example, the English word "can" can be a noun, an auxiliary verb, or a regular verb.

This research posits that while ambiguity is inherent in all languages, it is rarely identical across them. By leveraging parallel translations, the clarity of one language can resolve the ambiguity of another. This "triangulation" of linguistic structure allows for significantly more accurate unsupervised tagging, effectively halving the performance gap between unsupervised and supervised systems.

Methodology

The researchers developed two hierarchical Bayesian models that extend the standard Hidden Markov Model (HMM) framework to a multilingual context.

1. The Merged Node Model

Designed for bilingual pairs, this model merges aligned words from two languages into a single "bi-tag" node. It uses a "coupling parameter" to learn how tags in one language relate to those in another. While effective for pairs, this approach does not scale well; adding more languages causes the state space to explode exponentially.

2. The Latent Variable Model

To overcome scalability issues, the authors introduced a more sophisticated model using "superlingual tags." These latent variables act as a universal bridge between all languages in a parallel corpus.

  • Non-parametric Clustering: The model uses a Dirichlet Process to automatically determine the optimal number of superlingual clusters based on the data.
  • Product of Experts: Instead of simple averaging, the model uses a "product of experts" approach. If any language's transition model or the superlingual bridge assigns a low probability to a tag, that tag is unlikely to be selected, encouraging consensus across the multilingual group.
Comparison of (a) Monolingual HMMs, (b) the Merged Node Model, and (c) the Latent Variable Model.

Both models utilize Gibbs sampling for inference, iteratively updating tags to find the most probable global assignment.

Experimental Setup

The models were evaluated using the Multext-East corpus, which contains translations of George Orwell’s 1984 in eight languages: Bulgarian, Czech, English, Estonian, Hungarian, Romanian, Serbian, and Slovene. Automated word alignment was performed using GIZA++, providing the necessary cross-lingual anchors without human intervention.

Key Results and Insights

The multilingual approach yielded substantial performance improvements over monolingual baselines:

  • State-of-the-Art Accuracy: Using all eight languages, the Latent Variable Model reached an average accuracy of 95%, significantly outperforming the 91.1% monolingual baseline and nearing the 97.3% performance of supervised systems.
  • The Power of Eight: A striking finding was that performance improved consistently as more languages were added, with no observed plateau. The Latent Variable Model's linear complexity growth makes it a viable framework for massively multilingual learning.
  • Robustness in Data-Poor Scenarios: In tests with a reduced lexicon (knowing tags for only the 100 most frequent words), the multilingual model maintained 82.8% accuracy, a massive jump from the 74.8% monolingual baseline. This demonstrates effective "indirect supervision" through cross-lingual transfer.

Conclusion and Future Work

This work demonstrates that multilingual evidence can significantly ease the challenge of unsupervised POS tagging. However, the models remain dependent on the quality of automated word alignments and currently assume a predominantly one-to-one mapping between words across languages.

Future directions include incorporating explicit typological knowledge about language families to further refine superlingual clusters and scaling the framework to handle non-parallel data across dozens of languages.

Summary updated

Abstract

We demonstrate the effectiveness of multilingual learning for unsupervised part-of-speech tagging. The central assumption of our work is that by combining cues from multiple languages, the structure of each becomes more apparent. We consider two ways of applying this intuition to the problem of unsupervised part-of-speech tagging: a model that directly merges tag structures for a pair of languages into a single sequence and a second model which instead incorporates multilingual context using latent variables. Both approaches are formulated as hierarchical Bayesian models, using Markov Chain Monte Carlo sampling techniques for inference. Our results demonstrate that by incorporating multilingual evidence we can achieve impressive performance gains across a range of scenarios. We also found that performance improves steadily as the number of available languages increases.
BibTeX
@article{naseem2009multilingual,
  author = {Tahira Naseem and Benjamin Snyder and Jacob Eisenstein and Regina Barzilay},
  title = {{Multilingual Part-of-Speech Tagging: Two Unsupervised Approaches}},
  journal = {J. Artif. Intell. Res.},
  year = {2009},
  url = {https://openreview.net/forum?id=W5Y8HL9wiDc},
}
Figure