Decomposing Fusional Morphemes with Vector Embeddings

Michael Ginn, Alexis PalmerNAACL 2024

TL;DR

This paper explores whether vector embeddings, commonly used for word meaning, can effectively capture the complex grammatical information encoded within fusional morphemes. By training embeddings on morphological sequences from two low-resource languages, they demonstrate that these embeddings align with linguistic features, suggesting a promising approach for modeling morphology, especially in languages with complex inflectional systems.

What Problem Does This Paper Address?

Traditional word embeddings excel at capturing semantic and syntactic relationships by analyzing word co-occurrence patterns. However, morphology, the study of word formation and structure, presents a unique challenge. Specifically, "fusional morphemes" are single units that encode multiple grammatical features simultaneously (e.g., person, number, and case all in one affix). This complexity makes it difficult to disentangle the individual linguistic dimensions they represent. While distributional semantics has been applied to phonology, its effectiveness in modeling the intricate, multi-dimensional nature of fusional morphology has been less explored. This paper aims to bridge this gap by investigating if vector embeddings can learn and represent these complex morphological features.

Methods

The authors employ standard vector embedding techniques, specifically Singular Value Decomposition (SVD) and word2vec (both skip-gram and CBOW), to learn representations of morphological categories. Instead of embedding individual morphemes (which are often too rare in low-resource languages), they train embeddings on sequences of morphological glosses from interlinear glossed text (IGT). This approach treats sequences of glosses as the "context" for learning embeddings.

To evaluate their findings, they compare the learned embedding spaces to handcrafted linguistic feature spaces. For fusional morphemes, they create binary feature vectors based on linguistic dimensions (like case, person, number) from the UniMorph schema. They then measure the cosine similarity between gloss pairs in both the embedding space and the linguistic feature space. A key aspect of their evaluation is comparing these correlations to a baseline where random vectors are used in the embedding space, controlling for chance similarities.

Datasets

The study utilizes data from two low-resource languages:

  • Uspanteko (usp): An endangered Mayan language with approximately 6,000 speakers. The data is derived from Ginn et al. (2023) and consists of 9,774 sentences.
  • Tsez (ddo): A Nakh-Daghestanian language with around 14,000 speakers. The data is from the Tsez Annotated Corpus Project (Abdulaev et al., 2022) and includes 7,116 sentences.

The authors specifically use the gloss lines from IGT data, treating these sequences of morphological category glosses as the input for their embedding models.

Key Insights and Ideas

  • Vector embeddings can capture fusional morphology: The core finding is that distributional representations, learned from sequences of morphological glosses, can indeed capture meaningful linguistic dimensions encoded in fusional morphemes.
  • word2vec outperforms SVD for linguistic similarity: The word2vec models (both CBOW and skip-gram) showed a stronger tendency to group linguistically related glosses together compared to the SVD model. This suggests that the neural network architecture of word2vec is more adept at learning these nuanced relationships from limited data.
  • Context matters for morphological embeddings: Training on sequences of glosses (morphological context) rather than individual morphemes is crucial, especially for low-resource languages where individual morphemes are infrequent. This allows for the induction of meaningful representations.
  • Linguistic features are reflected in embedding geometry: The study provides quantitative evidence that the geometric relationships between glosses in the learned embedding space correlate with the relationships defined by manually constructed linguistic feature vectors. This validates the idea that distributional properties reflect underlying grammatical structure.
  • Low-resource languages benefit from this approach: The success with Uspanteko and Tsez demonstrates the viability of this method even when data is scarce, making it a valuable tool for linguistic documentation and analysis.

Key Results

  • Correlation with linguistic features: The study found statistically significant correlations between the embedding space similarities and the linguistic feature space similarities for both languages, with p < 0.001.
  • word2vec vs. SVD performance: While SVD achieved higher maximum correlations in some cases, word2vec models generally showed more consistent and reliable correlations across hyperparameters, indicating better capture of linguistic relationships.
  • Hyperparameter sensitivity: For SVD, smaller window sizes (1-2) yielded better correlations, suggesting that local context is key for capturing these specific linguistic features. Word2vec models showed more stable performance across window sizes.
  • Mean correlations are moderate: The mean correlations reported are described as "fairly low," which the authors attribute partly to the small dataset size and potentially to models learning relationships beyond the explicitly defined linguistic dimensions.

Limitations and Future Work

  • Limited language diversity: The study was conducted on only two languages. The authors acknowledge that results might vary across languages, particularly those with less fusional morphology.
  • Dataset size: The relatively small size of the datasets for these low-resource languages is noted as a potential factor limiting the strength of the observed correlations.
  • Future Work:
    • Broader language coverage: Investigating a wider range of languages to understand how fusional morphology varies and how embeddings adapt.
  • Deeper exploration of embedding spaces: Further analysis to uncover novel morphological relationships beyond those explicitly defined by the linguistic features.
  • Applications: Exploring practical applications such as improving morpheme glossing models or aiding linguistic analysis with data-driven evidence.

Summary updated

BibTeX
@inproceedings{ginn2024decomposing,
  author = {Michael Ginn and Alexis Palmer},
  title = {{Decomposing Fusional Morphemes with Vector Embeddings}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure