Sigmorphon 2019 Task 2 system description paper: Morphological analysis in context for many languages, with supervision from only a few

Brad Aiken, Jared Kelly, Alexis Palmer, Suleyman Olcay Polat, Taraka Rama, Rodney NielsenACL 2019

TL;DR

This paper introduces a novel approach to morphological analysis for many languages, requiring minimal labeled training data by leveraging parallel Bible corpora. The core idea is to align contextual word embeddings across languages using Bible verses and then project UniMorph tag representations to infer morphological features, aiming for broad language applicability.

What Problem Does This Paper Address?

Morphological analysis, the process of identifying word structure and grammatical features, is crucial for natural language processing. However, creating high-quality morphological analyzers typically requires extensive labeled training data, which is scarce for most of the world's languages. This paper tackles the challenge of building morphological analyzers for low-resource languages, aiming to develop a system that can be applied to languages with very little or no specific training data, relying instead on readily available resources like parallel Bible translations.

Methods

The UNT HiLT+Ling system's core contribution lies in its minimally-supervised approach to morphological analysis. The key steps are:

  • Contextual Embeddings: They use ELMo (Embeddings from Language Models) to generate contextual word embeddings for each language's Bible translation.
  • Verse-Level Alignment: To bridge the gap between languages, they align embedding spaces by treating Bible verses as parallel data. They compute verse embeddings as the average of token embeddings within a verse and then learn a translation matrix to map embeddings from a source language (English) to target languages. This is a novel application of alignment techniques to contextual embeddings, overcoming the challenge of dynamic embeddings.
  • UniMorph Tag Induction: Instead of relying on labeled data for morphological tags, they induce vector representations for UniMorph tags. This is done by creating sets of words with and without a specific tag in English and then projecting these tag vectors into target languages using the learned translation matrices.
  • Morphological Tagging: At test time, for a given word in context, they subtract the projected UniMorph tag vectors from the word's embedding. If the resulting vector is close to the embedding of a related word (e.g., subtracting the "plural" tag vector from "children" results in a vector close to "child"), the tag is assigned.

The system also includes secondary components for Part-of-Speech (POS) tagging (using an HMM) and lemmatization (using a seq2seq neural architecture), which utilize more standard supervised learning from the provided training data.

Datasets

  • Multilingual Bible Corpus: This is the primary dataset for building the core model. The authors used Bible translations for 41 languages from Christodouloupoulos and Steedman (2015) and sourced Bibles for an additional 19 languages. For languages without a Bible, they used "proxy languages" (closely related dialects or typologically similar languages).
  • UniMorph 2.0 Data-set: This dataset provides morphological tag sets for 111 languages and is used for defining the target morphological features.
  • Sigmorphon 2019 Task 2 Training and Development Data: This data is used for training the POS tagger and lemmatizer, and for building contrasting word sets for UniMorph tag induction.

Key Insights and Ideas

  • Bible as a Universal Linguistic Resource: The paper demonstrates the remarkable utility of the Bible as a parallel corpus for cross-lingual NLP tasks, especially for low-resource languages, due to its wide availability and inherent verse-level alignment.
  • Aligning Contextual Embeddings with Verse-Level Anchors: A key innovation is using verse-level alignment to map dynamic ELMo embeddings across languages, circumventing the need for large bilingual dictionaries.
  • Inducing Morphological Tags from Analogies: The idea of representing morphological tags as vectors and using analogy-like tasks to infer them is a clever way to reduce reliance on labeled tag data.
  • Minimal Supervision as a Design Principle: The overarching goal of making morphological analysis feasible for languages with virtually no annotated data is a significant and impactful research direction.

Key Results

The paper candidly reports that their system's performance on the morphological analysis task was "dramatically below the average system submitted." Specifically, their accuracies were "at least 50% worse than every other team, on most languages." This poor performance is attributed to several limitations, including the computational intensity of ELMo, shortcuts taken for submission, and the inherent challenges of their minimally-supervised approach. For lemmatization, their performance was closer to the average, about 12% less accurate than the top system. Table 3 shows that performance degrades when only the New Testament is used, but surprisingly, using proxy languages yields only slightly lower performance than using a direct Bible translation.

Limitations and Future Work

  • Computational Intensity: ELMo embeddings are computationally expensive, leading to shortcuts like storing tagged words and not processing all test data.
  • Tagging Constraints: The system doesn't constrain the set of possible tags, allowing for multiple conflicting tags to be assigned simultaneously.
  • Morphological Paradigms: The current method assumes inflected/uninflected word pairs, which doesn't account for all morphological paradigms (e.g., sets of inflectional options).
  • Out-of-Vocabulary (OOV) Tokens: The system performs poorly on OOV tokens.
  • Future Work: The authors suggest experimenting with verse-trained models, using all available Bible data for training, investigating new methods for identifying transfer languages, and exploring verse-level alignment for embedding spaces. They believe their architecture has potential for extending to the vast number of languages with Bible translations.

Summary updated

BibTeX
@inproceedings{aiken2019sigmorphon,
  author = {Brad Aiken and Jared Kelly and Alexis Palmer and Suleyman Olcay Polat and Taraka Rama and Rodney Nielsen},
  title = {{Sigmorphon 2019 Task 2 system description paper: Morphological analysis in context for many languages, with supervision from only a few}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}

Loading related content...

Figure