Unsupervised Part-of-Speech Acquisition for Resource-Scarce Languages

Sajib Dasgupta, Vincent NgEMNLP 2007

TL;DR

This paper introduces a novel bootstrapping approach for unsupervised Part-of-Speech (POS) tagging, specifically designed for languages with limited linguistic resources. By integrating morphological information with distributional analysis, the method significantly improves the quality of initial "seed" word clusters, leading to more accurate POS induction, even for highly inflectional languages.

What Problem Does This Paper Address?

Developing accurate Natural Language Processing (NLP) tools like POS taggers and parsers hinges on having high-quality lexicons. While resource-rich languages like English have readily available hand-crafted lexicons, this is not the case for "resource-scarce" languages. Manually creating lexicons is time-consuming, requires linguistic expertise, and is practically infeasible for languages with complex morphology (e.g., agglutinative languages). Supervised learning methods, which rely on annotated data, are thus limited. This paper tackles the challenge of acquiring POS information in an unsupervised manner, aiming to automate lexicon construction for languages with scarce resources. The core problem is how to effectively leverage available unannotated text to learn the possible POS tags for words without prior linguistic knowledge.

Methods

The paper proposes a three-step bootstrapping approach:

  1. Morphological Clustering: This is the core innovation. Instead of relying solely on distributional similarity (words appearing in similar contexts), the authors first cluster words based on shared morphological patterns (suffixes). They develop a method to identify "useful" suffixes and use them to create initial clusters. This step is crucial because it leverages inherent word structure, which can be more reliable than sparse distributional data, especially for morphologically rich languages. They use an unsupervised morphological analyzer to segment words and identify suffixes.
  2. Seed Set Purification: The initial morphological clusters might still contain mislabeled words (e.g., adjectives misclassified as nouns due to shared suffixes). This step purifies the seed set by identifying and removing words where morphological and distributional information disagree on their POS. They train a multi-class classifier (using SVM) on the morphologically clustered words and compare its predictions with the morphological labels. Words with conflicting predictions are flagged as potentially mislabeled. They also filter out words with very low corpus frequency (unreliable for distributional analysis).
  3. Bootstrapping (Augmenting the Seed Set): The purified seed set is then used to train a weakly supervised learner (a version of self-training with SVM). This classifier is applied to label unlabeled words. Words labeled with high confidence are added to the seed set, and the classifier is retrained iteratively. This process gradually expands the labeled data and refines the POS induction.

A key aspect is the integration of both morphological and distributional features, with SVM automatically weighting their importance.

Datasets

  • English: Wall Street Journal (WSJ) portion of the Penn Treebank.
  • Bengali: Five years of articles from the Bengali newspaper Prothom Alo.

The authors also created a manually annotated 5000-word Bengali lexicon for evaluation purposes, noting this as a contribution to the field due to the scarcity of such resources.

Key Insights and Ideas

  • Morphology as a powerful signal for POS: The paper strongly advocates for the integration of morphological information, arguing that it can be more reliable than purely distributional cues, especially for morphologically rich languages where distributional data might be sparse or misleading.
  • Bridging the gap between morphology and distribution: The method's strength lies in its ability to combine these two information sources, using morphology to create initial, high-quality clusters and then using distribution to refine and purify them.
  • Unsupervised morphological analysis is feasible: The reliance on an unsupervised morphological analyzer that works well for both English and Bengali demonstrates that such tools can be built even without extensive language-specific resources.
  • Purification is critical: The purification step, where conflicting morphological and distributional signals are resolved, significantly improves the accuracy of the seed set, leading to better bootstrapping results.
  • Applicability to diverse language types: The success on both English (morphologically impoverished) and Bengali (highly inflectional) suggests the robustness and broad applicability of the proposed approach.

Key Results

  • The proposed unsupervised POS induction system significantly outperforms a strong baseline (Clark's cluster_neyessenmorph) on both English and Bengali.
  • For English, the system achieves an F1-score of 73% compared to the baseline's 59% after bootstrapping.
  • For Bengali, the system achieves an F1-score of 76% compared to the baseline's 45% after bootstrapping.
  • The purification step is shown to be particularly effective, boosting F1-scores substantially.
  • The method also performs well on rare words, which are often challenging for distributional methods.

Limitations and Future Work

  • Ambiguity: While the system handles some ambiguity, it acknowledges that ideally, a "soft clustering" approach would be more suitable for lexical ambiguity. Their attempt to incorporate soft clustering in bootstrapping did not yield better results.
  • Bengali Proper Nouns: The paper notes that distinguishing Bengali proper nouns from common nouns remains a challenge, suggesting it as a topic for future research.
  • Future work: The authors suggest further research into handling Bengali proper nouns and exploring more sophisticated methods for soft clustering.

Summary updated

BibTeX
@inproceedings{dasgupta2007unsupervised,
  author = {Sajib Dasgupta and Vincent Ng},
  title = {{Unsupervised Part-of-Speech Acquisition for Resource-Scarce Languages}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2007},
}
Figure