Data Augmentation for Context-Sensitive Neural Lemmatization Using Inflection Tables and Raw Text

Toms Bergmanis, Sharon GoldwaterNAACL 2019

TL;DR

This paper tackles the challenge of training accurate context-sensitive neural lemmatizers for low-resource languages, where large annotated datasets are scarce. They propose a novel data augmentation technique that leverages readily available inflection tables and raw text from Wikipedia to create valuable training examples, significantly improving lemmatization accuracy, especially for unseen words.

What Problem Does This Paper Address?

Lemmatization, the process of reducing inflected word forms to their base dictionary form (lemma), is crucial for many Natural Language Processing (NLP) tasks. While context-sensitive lemmatizers can handle ambiguous and unseen words better than isolated-word approaches, they typically require extensive, fully annotated sentence data for training. This is a major bottleneck for low-resource languages where such data is rare or non-existent. The authors highlight that for scarce data, training on a larger number of distinct word types is more beneficial than training on a similar number of word tokens, as tokens often repeat types. They aim to bridge this gap by developing a method to train effective context-sensitive lemmatizers with minimal labeled data.

Methods

The core of this paper is a data augmentation strategy that combines two key sources:

  • UniMorph Inflection Tables: These tables provide lemma-inflection pairs for various word categories (nouns, verbs, adjectives). The authors extract unambiguous lemma-inflection pairs from these tables.
  • Wikipedia Raw Text: For each unambiguous pair, they find sentences in Wikipedia that contain the inflected form. This provides the context for the word.

This process creates synthetic training examples consisting of an inflected wordform within its sentence context, paired with its lemma. The authors then use these augmented examples to train a neural sequence-to-sequence model called Lematus (which uses an N-character context window). They compare this approach to baselines, including a context-free lemmatizer (Lemming) and a context-sensitive model trained on limited data (HMAM), as well as a simple "most frequent lemma" baseline. A key aspect is that Lematus does not require fully annotated sentences, only wordforms with their local context.

Datasets

  • Universal Dependency Treebank (UDT): Used for initial training data (1k types) and as the primary evaluation benchmark across 10 languages.
  • UniMorph Project: Provides inflection tables for generating augmented training data.
  • Wikipedia: Used as a source of raw text to extract sentence contexts for the augmented examples.

Key Insights and Ideas

  • Type-based learning is crucial for low-resource settings: When data is scarce, seeing more distinct word types provides more morphological information than seeing more tokens of the same few types.
  • Context can help even with unambiguous data: While the augmented examples are derived from unambiguous forms, the context still proves beneficial for the lemmatizer, especially for unseen words. This challenges the intuition that context is only useful for disambiguation.
  • Noisy but useful augmentation: The UniMorph + Wikipedia data is acknowledged as noisy and potentially biased (e.g., limited POS coverage, annotation inconsistencies), yet it still leads to significant improvements.
  • Contextual cues are learned even from simple examples: The success of Lematus 20-ch (with context) over Lematus 0-ch (without context) even with limited augmented data suggests that the model can extract valuable contextual information from relatively simple, unambiguous examples.

Key Results

  • Significant improvements on unseen words: The proposed data augmentation method consistently improves lemmatization accuracy, particularly for unseen words, which are a major challenge in low-resource scenarios.
  • Contextual models benefit more: While all systems see some improvement with augmentation, context-sensitive models like Lematus 20-ch show more pronounced gains, especially when compared to their context-free counterparts (Lematus 0-ch).
  • Type-based training outperforms token-based in low-resource settings: For very limited data (1k examples), training on types yields better accuracy than training on tokens for most systems.
  • Augmentation helps across different resource levels: The method shows benefits in both very low-resource (1k types) and medium-resource (10k tokens) settings. For instance, adding 10k UM types to 10k UDT tokens significantly boosts accuracy.
  • Specific language gains: Tables show consistent improvements across all 10 tested languages, with some languages showing larger gains than others. For example, in Table 3, Lematus 20-ch with 10k UM types achieves 70.9% accuracy on unseen words for Russian, a substantial jump.

Limitations and Future Work

  • Data Bias and Coverage: The UniMorph tables have limited coverage (nouns, verbs, adjectives) and potential annotation inconsistencies, which can affect the quality of augmented data. This leads to lower accuracy compared to models trained on larger, more comprehensive datasets like UDT.
  • Ambiguity Handling: The method relies on selecting unambiguous forms from UniMorph. Handling truly ambiguous forms would require further advancements.
  • POS Coverage: The limited POS coverage in UniMorph means that augmented data is less effective for parts of speech not included in the tables, as observed in the Latvian error analysis.
  • Future Work: The authors suggest exploring ways to mitigate the issues of biased/incomplete data and to extend the approach to handle more parts of speech. Further investigation into the specific types of errors made by the augmented models could also guide future improvements.

Summary updated

BibTeX
@inproceedings{bergmanis2019data,
  author = {Toms Bergmanis and Sharon Goldwater},
  title = {{Data Augmentation for Context-Sensitive Neural Lemmatization Using Inflection Tables and Raw Text}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure