A Backoff Model for Bootstrapping Resources for Non-English Languages

Chenhai Xi, Rebecca HwaEMNLP 2005

TL;DR

This paper tackles the challenge of building NLP tools for languages with scarce annotated data by proposing a novel "backoff" model. This model effectively combines a small amount of manually annotated data with a larger, automatically generated dataset derived from English resources, significantly outperforming existing bootstrapping methods.

What Problem Does This Paper Address?

Developing high-performing Natural Language Processing (NLP) applications, especially those relying on supervised learning, hinges on the availability of large, annotated datasets. This is a well-known bottleneck, but the problem is particularly acute for non-English languages, where such resources are even scarcer. While "bootstrapping" techniques, which leverage existing high-quality English NLP tools and parallel corpora (texts translated between languages), have been explored, their effectiveness is limited, especially for language pairs with significant structural differences (like English and Chinese). These projection-based methods often suffer from noisy automatically generated annotations due to imperfect word alignments and linguistic dissimilarities. This paper aims to bridge this gap by proposing a method that makes better use of both limited manual annotations and noisy projected data.

Methods

The core contribution of this paper is a novel backoff model for combining two distinct data sources:

  1. Projected Data (Tproj): This is a large corpus of Chinese text automatically annotated with Part-of-Speech (POS) tags. These tags are derived by projecting POS tags from aligned English words in a parallel English-Chinese corpus. The authors adapt existing projection techniques to handle challenges like one-to-many word alignments and differing tag sets between languages.
  2. Manually Annotated Data (Tanno): This is a small, carefully selected set of Chinese sentences that have been manually annotated with POS tags. The selection process uses a Maximum Word Coverage (MWC) algorithm, inspired by active learning, to prioritize sentences that introduce the most new vocabulary to the model, thereby reducing the problem of unknown words.

The proposed backoff model (Tback) combines these two sources by using the manually annotated data as the primary source for estimating POS tagger parameters (both transition and observation probabilities). When a particular word-tag combination is not seen in the manual data, it "backs off" to use the parameters derived from the larger, projected dataset. This approach is designed to leverage the broad word coverage of the projected data while relying on the higher quality of the manual annotations for reliable tag transitions and observations.

Datasets

  • English-Chinese Federal Broadcast Information Service (FBIS) corpus: Used as the source for the parallel corpus to generate projected annotations.
  • Chinese Treebank version 4 (CHTB): Used to simulate manual annotations. A subset of CHTB was used as held-out data, and the remaining sentences were split into ten-fold cross-validation sets. Approximately 12,600 sentences were used for training the projected data tagger (Tproj), and a subset of this was used for manual annotation selection.

Key Insights and Ideas

  • Projected data is noisy but valuable: While projection methods create noisy annotations, they still contain useful information, especially for improving word coverage, which is crucial for POS tagging.
  • Manual annotation is scarce, so make it count: The MWC algorithm highlights the importance of strategically selecting a small set of manual annotations to maximize their impact, particularly in reducing the number of unknown words.
  • Backoff is better than simple interpolation or concatenation: Directly interpolating or concatenating models trained on noisy projected data and clean manual data can degrade performance. A backoff strategy, which prioritizes cleaner data and uses noisier data as a fallback, proves more effective.
  • Language dissimilarity is a major hurdle: The limited success of projection techniques for English-Chinese compared to English-French suggests that greater linguistic divergence significantly impacts the quality of projected annotations.

Key Results

  • A tagger trained solely on projected data (Tproj) achieved an accuracy of 64.5% after noise reduction and filtering.
  • A tagger trained only on manually annotated data (Tanno) achieved 74.3% accuracy with 30,000 manually annotated tokens.
  • The proposed backoff model (Tback) achieved 74.9% accuracy when initialized with a backoff tagger using 3000 manually annotated tokens, outperforming Tanno and other semi-supervised methods like self-training and EM, especially when manual annotations were severely limited.
  • The backoff approach showed significant improvement over Tanno when fewer than 10,000 manually annotated tokens were available.

Limitations and Future Work

  • Limitations: The paper acknowledges that the backoff model is an approximation and that other methods for estimating merged HMM parameters might exist. The performance of Tmerge (which uses projected observation probabilities and manual transition probabilities) was worse than Tanno, suggesting that observation probabilities from manual data are critical.
  • Future Work: The authors suggest exploring the application of their backoff approach to other NLP tasks and models, comparing it with more diverse multi-learner combination methods, and adapting it to more complex tasks like parsing. They also propose that their backoff tagger could serve as a better initialization for bootstrapping algorithms like self-training, which they demonstrated with a slight improvement.

Summary updated

BibTeX
@inproceedings{xi2005backoff,
  author = {Chenhai Xi and Rebecca Hwa},
  title = {{A Backoff Model for Bootstrapping Resources for Non-English Languages}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2005},
}
Figure