Multilingual Part-of-Speech Tagging: Two Unsupervised Approaches

Tahira Naseem, Benjamin Snyder, Jacob Eisenstein, Regina Barzilay

TL;DR

This paper demonstrates that the "headache" of unsupervised part-of-speech (POS) tagging can be significantly eased by looking at multiple languages at once. By using a non-parametric Bayesian model to "triangulate" linguistic structure across eight different languages, the researchers achieved up to 95% accuracy—effectively halving the performance gap between unsupervised and supervised systems.

What Problem Does This Paper Address?

Part-of-speech tagging—assigning labels like "noun," "verb," or "adjective" to words—is a foundational task in Natural Language Processing (NLP). While supervised systems are highly accurate, they require massive amounts of human-annotated data, which doesn't exist for most of the world's 7,000+ languages.

Unsupervised tagging tries to learn these patterns from raw text alone, but it faces a major hurdle: ambiguity. In English, the word "can" could be a noun (a tin can), an auxiliary verb (I can go), or a regular verb (to can peaches). A monolingual model has to guess based on local context. However, in another language, these three senses might be expressed by three entirely different words. The authors argue that by looking at parallel translations, the ambiguity in one language can be resolved by the clarity in another.

Methods

The authors propose two hierarchical Bayesian models that build upon the standard Hidden Markov Model (HMM) structure:

  1. The Merged Node Model: Designed for language pairs, this model merges aligned words into a single "bi-tag" node. It uses a "coupling parameter" to learn how tags in Language A relate to tags in Language B. While effective, it doesn't scale well; adding a third or fourth language causes the state space to explode exponentially.
  2. The Latent Variable Model: This is the more sophisticated, scalable approach. It introduces "superlingual tags"—latent variables that act as a bridge between all languages in a parallel corpus.
  3. Non-parametric clustering: It uses a Dirichlet Process to automatically determine how many of these superlingual clusters are needed based on the data.
  4. Product of Experts: Instead of a simple average, the model uses a "product of experts" approach. If any one language's transition model or the superlingual bridge "vetoes" a tag (assigns it low probability), that tag is unlikely to be chosen. This encourages consensus across the multilingual group.

For both models, the authors used Gibbs sampling for inference, a technique that iteratively updates the tags to find the most probable global assignment.

Datasets

The researchers used the Multext-East corpus, which contains translations of George Orwell’s 1984 in eight languages: Bulgarian, Czech, English, Estonian, Hungarian, Romanian, Serbian, and Slovene.

To connect the languages, they used GIZA++, a standard tool for automated word alignment. This allowed the models to know which word in the English text corresponded to which word in the Hungarian text, even without human intervention.

Key Insights and Ideas

  • Ambiguity is Language-Specific: The core "aha" moment is that while all languages are ambiguous, they aren't ambiguous in the same way. Combining them filters out the noise.
  • The Power of Eight: One of the most striking findings was that performance didn't plateau. As the researchers added more languages to the Latent Variable Model, the accuracy for every language continued to improve.
  • Scaling Gracefully: The Latent Variable Model's complexity grows linearly with the number of languages, unlike the Merged Node Model. This makes it a viable framework for "massively multilingual" learning.
  • Robustness to Lexicon Size: Even when the model was given a very restricted dictionary (only the 100 most frequent words), the multilingual evidence provided enough "indirect supervision" to keep accuracy high.

Key Results

  • Top Performance: Using all eight languages and a full lexicon, the Latent Variable Model reached 95% average accuracy. For comparison, the monolingual baseline was 91.1%, and a fully supervised system reached 97.3%.
  • Closing the Gap: The multilingual approach halved the distance between "learning from scratch" and "learning from humans."
  • Significant Gains in Difficult Scenarios: In a "reduced lexicon" test (where the model only knows tags for frequent words), the multilingual model hit 82.8% accuracy, a massive jump from the 74.8% monolingual baseline.
  • Cross-Lingual Transfer: The authors showed that a "resource-rich" language (like English) could effectively "teach" a "resource-poor" language through the shared latent structure.

Limitations and Future Work

  • Alignment Dependency: The models rely heavily on the quality of word alignments. If the automated alignment (GIZA++) is messy, the "anchors" for the multilingual learning become unreliable.
  • One-to-One Assumption: The current models primarily focus on one-to-one word alignments. However, many languages use multiple words to express a concept that another language captures in a single morpheme (e.g., English "will go" vs. a single future-tense verb in Spanish).
  • Future Directions: The authors suggest incorporating more "typological" knowledge—explicitly telling the model about the grammatical properties of language families—to further refine the superlingual clusters. They also envision scaling this to dozens of languages using non-parallel data.

Summary updated

Abstract

We demonstrate the effectiveness of multilingual learning for unsupervised part-of-speech tagging. The central assumption of our work is that by combining cues from multiple languages, the structure of each becomes more apparent. We consider two ways of applying this intuition to the problem of unsupervised part-of-speech tagging: a model that directly merges tag structures for a pair of languages into a single sequence and a second model which instead incorporates multilingual context using latent variables. Both approaches are formulated as hierarchical Bayesian models, using Markov Chain Monte Carlo sampling techniques for inference. Our results demonstrate that by incorporating multilingual evidence we can achieve impressive performance gains across a range of scenarios. We also found that performance improves steadily as the number of available languages increases.
BibTeX
@article{naseem2009multilingual,
  author = {Tahira Naseem and Benjamin Snyder and Jacob Eisenstein and Regina Barzilay},
  title = {{Multilingual Part-of-Speech Tagging: Two Unsupervised Approaches}},
  journal = {J. Artif. Intell. Res.},
  year = {2009},
  url = {https://openreview.net/forum?id=W5Y8HL9wiDc},
}

Loading related content...

Figure