Using Language Modeling to Select Useful Annotation Data

Dmitriy Dligach, Martha PalmerNAACL 2009

TL;DR

This paper tackles the challenge of imbalanced datasets in natural language processing, specifically for Word Sense Disambiguation (WSD). They propose a novel, unsupervised method called Language Modeling for Data Selection (LMS) that leverages language models to identify and prioritize instances of rare word senses for annotation, significantly outperforming random sampling and K-means clustering.

What Problem Does This Paper Address?

Annotation is a crucial but expensive step in building NLP resources. Many NLP tasks, like Word Sense Disambiguation (WSD), suffer from "imbalanced data." This means that for a given word, one or a few senses are extremely common, while many other senses are very rare. This skew is problematic because standard machine learning models tend to perform poorly on rare classes. If an annotation project has limited resources and can only annotate a fraction of available data, simply picking instances randomly will likely result in a dataset that still heavily favors the common senses, leaving the rare ones underrepresented. This is a significant issue because rare senses, while infrequent in a specific corpus, might be important in other domains or for specific applications. The paper addresses the question of how to intelligently select a subset of data for annotation that maximizes the representation of these rare, but potentially important, word senses.

Methods

The core contribution is the Language Modeling for Data Selection (LMS) method. The intuition behind LMS is that instances of rare word senses will likely have lower probabilities when modeled by a language model trained on the entire corpus. Here's how it works:

  1. Language Model Creation: A language model is trained on a large corpus.
  2. Instance Probability Calculation: For a target ambiguous word, each potential instance (typically a sentence or a chunk of text centered on the word) is assigned a probability by the language model. To account for varying instance lengths, the log probability is normalized by the number of words in the instance, a metric related to perplexity.
  3. Ranking and Selection: Instances are ranked based on their normalized log probability (lower probability indicates higher likelihood of being a rare sense instance). The instances with the lowest probabilities are then selected for annotation.

This method is unsupervised, requiring only a language modeling toolkit (like SRILM) and a corpus. It contrasts with active learning, which requires some initial labeled data and can be slow due to iterative labeling and pre-processing steps.

Datasets

The paper uses the OntoNotes dataset for evaluation. Specifically, they select verbs from OntoNotes that have at least 100 instances and where the rare sense constitutes less than 20% of the data. They evaluate on 11 such verbs, totaling 2,230 instances. No new datasets or benchmarks are introduced.

Key Insights and Ideas

  • Language Models as "Sense Detectors": The paper demonstrates that language models, typically used for text generation or prediction, can implicitly capture information about word sense distributions. Lower probability sequences often correspond to less common linguistic phenomena, including rare word senses.
  • Unsupervised Advantage: LMS's unsupervised nature is a significant advantage. It doesn't require any pre-annotated data to start, making it applicable even when no labeled data exists for a particular word.
  • Efficiency for Annotation: By prioritizing instances likely to contain rare senses, LMS can help annotation projects achieve better sense coverage with limited resources, potentially saving significant time and cost compared to random sampling.
  • LMS vs. Clustering: While K-means clustering is a standard technique for grouping data, LMS shows a more direct and effective way to target rare instances by leveraging the probabilistic nature of language models.

Key Results

  • Outperforms Random Sampling: LMS consistently outperformed random sampling in terms of recall for rare senses. On average, LMS achieved 72% recall for rare senses in the selected cluster, compared to 50% for random sampling when selecting half the data. For some verbs like "compare" and "point," LMS achieved 100% recall of rare senses.
  • Outperforms K-means Clustering: LMS also showed a favorable comparison to K-means clustering. Despite K-means having access to richer linguistic features and some seeded data, LMS achieved an average of 3% better recall for rare senses.
  • Plausibility Demonstrated: Initial experiments on verbs like "compare" and "add" visually demonstrated that LMS effectively concentrates rare sense instances in the lower probability clusters.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that K-means was given an advantage by having its cluster sizes dictated by the algorithm, while LMS's cluster size was set to match.
  • K-means used richer linguistic features and some seeded data, which LMS did not.
  • Future Work:
    • Investigate the impact of LMS-selected data on the performance of WSD models learned from it.
  • Apply LMS for domain adaptation, where a language model trained on one corpus is used to select data from another.
  • Explore other outlier detection techniques for data selection.
  • Develop a simplified active learning approach (single iteration) and compare it to LMS.
  • Investigate LMS-selected data as an initial training set for active learning, especially for pseudo-monosemous verbs (verbs with multiple senses defined but only one appearing in data).

Summary updated

BibTeX
@inproceedings{dligach2009using,
  author = {Dmitriy Dligach and Martha Palmer},
  title = {{Using Language Modeling to Select Useful Annotation Data}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2009},
}

Loading related content...

Figure