Orthographic vs. Semantic Representations for Unsupervised Morphological Paradigm Clustering

E. Margaret Perkoff, Josh Daniels, Alexis PalmerACL 2021

TL;DR

This paper explores unsupervised methods for grouping related word forms (like "walk," "walks," "walking") into their respective morphological paradigms. They found that character-based representations, focusing on spelling, generally outperform semantic (Word2Vec) embeddings for this task, especially for languages with regular morphology.

What Problem Does This Paper Address?

Morphological analysis, the study of word structure and formation, is crucial for understanding languages. However, creating robust automated systems is hampered by the scarcity of labeled data for most of the world's languages. This "low-resource" problem makes unsupervised learning, which doesn't require pre-labeled data, a vital area of research. Specifically, this paper tackles the challenge of unsupervised morphological paradigm clustering: grouping together all the different forms of a single word (e.g., "run," "runs," "ran," "running") without any prior knowledge of the language's grammar. This is a key step towards building better language processing tools for under-resourced languages.

Methods

The core of the proposed system is a KMeans clustering algorithm. The novelty lies in the representations of the words fed into this algorithm. The authors experiment with two main types:

  • Character-Based Embeddings: This method represents words based on their spelling. Each character in the language's alphabet is assigned a numerical value, and words are encoded as sequences of these values, padded to a uniform length. This approach prioritizes orthographic similarity.
  • Word2Vec Embeddings: This standard technique uses pre-trained (or trained on the corpus) word embeddings that capture semantic and syntactic relationships between words.

They also explore two strategies for initializing the KMeans cluster centers:

  • Randomly Initialized Centers: The algorithm starts with arbitrary cluster centers.
  • Pre-defined Centers (Optional/Future Work): This involves identifying potential "lemma" words (the base form) using methods like the Longest Common Substring (LCS) algorithm or graph-based approaches. These identified lemmas would then serve as the initial cluster centers.

Datasets

The authors used development languages (Maltese, Persian, Portuguese, Russian, and Swedish) for initial experiments and evaluation against gold-standard paradigm clusters. They then applied their best-performing models to a set of test languages including Basque, Bulgarian, English, Finnish, German, Kannada, Navajo, Spanish, and Turkish. The data for these languages was tokenized Bible text from the JHU Bible corpus.

Key Insights and Ideas

  • Spelling Matters (A Lot): For morphological paradigm clustering, orthographic similarity (how words are spelled) is surprisingly effective, often outperforming semantic similarity. This is particularly true for regular verb forms where spelling changes are predictable.
  • Irregularity is Hard: Both character-based and semantic representations struggle with irregular verb forms (e.g., "go," "went," "gone"). This highlights a significant challenge for unsupervised methods.
  • Character Embeddings are Robust for Regular Forms: The character-based approach successfully groups many regular verb forms, suggesting it's well-suited for synthetic and agglutinating languages where morphology is closely tied to spelling.
  • Pre-defined Centers Could Be a Game-Changer: While not fully implemented for evaluation in this paper, the idea of using methods like LCS to identify potential lemma words for cluster initialization holds significant promise for improving performance and avoiding the arbitrary definition of the number of clusters.

Key Results

  • Character Embeddings Dominate: For the development languages, KMeans with character embeddings (KMCE) consistently outperformed KMeans with Word2Vec embeddings (KMW2V) across all languages.
  • Baseline Beaten in Several Test Languages: KMCE models outperformed the baseline (which groups words by shared substrings) on Basque, German, Kannada, and Navajo.
  • Performance Varies with Cluster Size: For some test languages, increasing the number of clusters led to improved accuracy, suggesting that the optimal number of paradigms was not reached.
  • Word Length Can Be a Pitfall: Character-based representations, when padded to a fixed length, can sometimes group words of similar lengths together, even if they aren't morphologically related.

Limitations and Future Work

  • Irregular Forms Remain a Challenge: Neither representation method is currently robust enough to handle irregular verb forms effectively.
  • Character Distance Arbitrariness: The arbitrary mapping of characters to numerical values in character embeddings can lead to unexpected groupings if character distances don't reflect linguistic relationships.
  • Word Length Bias: The fixed-length vector representation can lead to errors based on word length.
  • Computational Cost of LCS Graphs: The LCS graph analysis for lemma identification was computationally too expensive for the current task.
  • Future Directions: The authors suggest exploring subword distances or cosine similarity for character representations, improving lemma identification methods (e.g., using LCS graphs more efficiently, exploring longest common sequences), incorporating other linguistic information (phonetics, part-of-speech), and filtering clusters based on language-specific properties. They also propose investigating hierarchical models that combine different algorithm types.

Summary updated

BibTeX
@inproceedings{perkoff2021orthographic,
  author = {E. Margaret Perkoff and Josh Daniels and Alexis Palmer},
  title = {{Orthographic vs. Semantic Representations for Unsupervised Morphological Paradigm Clustering}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure