Linguistic Problems in Multilingual Morphological Decomposition

G. ThurmairCOLING 1984

TL;DR

This paper presents a language-independent algorithm for morphological decomposition, breaking words down into their constituent morphemes. This is crucial for information retrieval systems, enabling them to find morphologically related terms and significantly improving search result relevance across multiple languages.

What Problem Does This Paper Address?

In information retrieval, users often struggle to find relevant documents because they don't know the exact form of keywords (e.g., singular vs. plural, different verb conjugations). Existing systems typically rely on exact keyword matching, leading to poor and meager search results. To overcome this, the authors propose a system that expands search queries morphologically, aiming to identify all words with the same root as the search term. This requires a robust method to decompose words into their fundamental meaningful units (morphemes) and then map these back to the original words. A key challenge is doing this effectively across multiple languages, each with its own unique linguistic rules and complexities.

Methods

The core of the system is a language-independent morphological decomposition algorithm. It operates using a morpheme lexicon (a list of known morphemes) and a morphological grammar for each language.

  1. Preprocessing: A language-specific step transforms the input string by handling simple grapheme insertions, deletions, and changes to eliminate some allomorphs (variant forms of a morpheme).
  2. Parsing: The transformed string is then processed by a parser that works in a left-to-right, breadth-first manner. It builds a network graph of possible morphemes.
  3. Morpheme Identification: At each step, the algorithm looks up potential morphemes in the lexicon and checks their validity using the morphological grammar. This grammar is represented as a state-transition automaton, defining allowed sequences of morpheme classes.
  4. Scoring and Selection: When multiple decompositions are possible (due to linguistic ambiguity), the system employs selection strategies. It prioritizes decompositions based on:
  5. Morpheme Class Sequences: Favoring more common patterns like suffixation over compounding.
  6. Longest Matching: If class sequences are ambiguous, it selects the decomposition with the fewest morphemes.
  7. Root Extraction and Inversion: The scored decomposition results are used to extract roots. Allomorphs of the same morpheme are mapped to the same set of words, ensuring that searching with any variant retrieves all related terms.

Datasets

The system was developed and tested on substantial datasets from three languages:

  • English: 36,000 tokens from the Food Science Technology Abstracts document files.
  • German: 53,000 tokens from the German Patent Office document files.
  • Spanish: 35,000 tokens from various text types (short stories, newspapers, etc.).

The paper does not explicitly mention newly introduced benchmarks, but the described datasets served as the basis for evaluating the system's performance.

Key Insights and Ideas

  • Language Independence is Achievable (Mostly): The core parsing algorithm is designed to be language-independent, with language-specific knowledge confined to the morpheme lexicon and grammar. This modularity is key to handling multiple languages.
  • The Ambiguity of "Morpheme": Defining what constitutes a morpheme is complex. The authors had to make pragmatic decisions, including frequent foreign words and derivations, and often mixing morphologies from different languages to capture synchronic usage.
  • Segmentation is Tricky: Deciding where one morpheme ends and another begins can be challenging. The authors opted for a semantic criterion to distinguish root meanings and distributional analysis, sometimes leading to compromises.
  • Allomorphs are Everywhere: Variant forms of morphemes (allomorphs) are common and need careful handling, either through preprocessing or by attaching information to the lexical entry.
  • Word vs. Morpheme as the Basic Unit: The research suggests that semantically and syntactically regular behavior is more evident at the word level than at the morpheme level, potentially challenging the idea of morphemes as the fundamental "basic units of meaning."

Key Results

  • High Root Extraction Accuracy: The system achieved correct root extraction in 95-97% of tokens across the tested languages.
  • Significant Retrieval Improvement: Retrieval results were improved by an overall 70% for the English version (German version was under testing).
  • Disambiguation Success: The disambiguation strategies returned correct roots in 90-94% of cases.
  • Overall Word Accuracy: Considering both single-result and disambiguated decompositions, the system worked correctly for approximately 96% of input words.
  • Error Rates: Error rates (words that could not be decomposed) were 4-8%, primarily due to proper names or typing errors.

Limitations and Future Work

  • Parts of Speech Derivation: It's difficult to reliably derive the part of speech of a word solely from its morphological decomposition. This is due to historical linguistic developments and inconsistencies (e.g., the German prefix "VER-" can form nouns, verbs, or adjectives).
  • Semantic Prediction: Predicting a word's meaning directly from its morphemes is not straightforward. The semantic formation rules don't always align with morphological ones, suggesting that words, rather than morphemes, might be the more fundamental units for semantic analysis.
  • Language-Specific Procedures: While the core algorithm is language-independent, the procedures attached to the grammar transitions (which handle specific constraints) are highly language-specific, posing a challenge to true language-independency.
  • Future Work: The authors suggest that further research is needed to integrate morphological decomposition into broader natural language systems and to explore more sophisticated disambiguation strategies, especially for languages with complex affix systems. They also highlight the need for more robust methods to derive semantic information and parts of speech.

Summary updated

BibTeX
@inproceedings{thurmair1984linguistic,
  author = {G. Thurmair},
  title = {{Linguistic Problems in Multilingual Morphological Decomposition}},
  booktitle = {International Conference on Computational Linguistics},
  year = {1984},
}
Figure