Automated Paradigm Selection for FSA based Konkani Verb Morphological Analyzer

Shilpa Desai, Jyoti Pawar, Pushpak BhattacharyyaCOLING 2012

TL;DR

This paper introduces a novel corpus-assisted Finite State Automata (FSA) approach for building a Konkani verb morphological analyzer. By leveraging an untagged corpus to map verb citation forms to paradigms, the authors significantly reduce human effort and achieve a high F-Score of 0.95, demonstrating an efficient method for developing morphological analyzers for morphologically rich languages.

What Problem Does This Paper Address?

Building morphological analyzers for morphologically rich languages, like Konkani, is a complex and time-consuming task. Traditional methods often rely on rule-based systems or require extensive manual input from linguistic experts to define paradigms and map words to them. This manual process is a bottleneck, especially when dealing with languages that have a large number of verb forms and complex inflectional systems. The paper addresses the challenge of automating and streamlining this paradigm selection process, aiming to reduce the human effort and time required to create accurate morphological analyzers.

Methods

The core of the paper's contribution lies in its "Corpus assisted FSA approach" for building a Verb Morphological Analyzer. Instead of sequencing characters, they use an FSA to sequence morphemes. The key innovation is the paradigm definition methodology and the use of a corpus for mapping verb citation forms to these paradigms.

Here's a breakdown of their approach:

  • Custom Paradigm Structure: They define their own paradigm structure for Konkani verbs, which uses FSA-based sequencing of morphemes and suffix classes. This allows for a compact representation of paradigms.
  • Suffix Classes: Suffixes are grouped into classes, simplifying the process. If one suffix from a class can attach to a stem, others in the same class can too.
  • Ambiguity Resolution with Corpus: Konkani verbs can exhibit stem formation ambiguities. To resolve this, they use an untagged corpus. When a verb citation form can map to multiple paradigms, they generate word variants for each potential paradigm using the FSA and then search for these variants in the corpus. The paradigm that yields the maximum number of found variants is selected as the correct one.
  • Two Main Modules:
    • FSA Module: This module handles the sequencing of morphemes for both analysis and generation of verb forms.
  • Paradigm Selector Module: This module maps an input verb to a paradigm, especially for verbs not already in the repository. It uses the corpus-driven approach described above.

Datasets

The paper utilizes the following datasets:

  • Konkani WordNet: Used to generate the verb lexicon.
  • Asmitai Corpus: An untagged corpus of approximately 268,000 words, primarily from the health and tourism domain. This corpus is crucial for the paradigm selection process.
  • ILCI Corpus: A tagged health and tourism domain corpus, used for evaluating the FSA module. Approximately 8697 unique verb variant forms were extracted from this corpus.

Key Insights and Ideas

  • Corpus as a Linguistic Expert: The paper demonstrates that an untagged corpus can effectively act as a proxy for human linguistic expertise in resolving morphological ambiguities and mapping words to paradigms. This is a significant step towards automating linguistic analysis.
  • Reducing Human Effort: The proposed method substantially reduces the manual effort required from linguists, making the development of morphological analyzers more scalable and efficient.
  • Handling Morphological Richness: The approach is particularly well-suited for morphologically rich languages like Konkani, where simple rule-based systems struggle with the complexity of inflections.
  • FSA for Morpheme Sequencing: Using FSA for morpheme sequencing, rather than just character sequencing, is a more direct and efficient way to model verb morphology.
  • Identifying Data Quality Issues: The FSA module's analysis of the tagged corpus also revealed spelling and tagging errors, highlighting its potential for corpus validation.

Key Results

The paper reports impressive results for both modules:

  • Paradigm Selector Module:
    • Achieved a Precision of 0.93, Recall of 0.67, and F-Score of 0.78.
  • Successfully mapped paradigms for 1003 out of 1226 verbs.
  • FSA Module:
    • Achieved a Precision of 0.992, Recall of 0.919, and F-Score of 0.954 when tested on a tagged corpus.
  • Successfully analyzed 7237 out of 8697 unique verb variants.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the recall of the paradigm selector module could be improved by augmenting the corpus with more verb forms.
  • The FSA module's recall can be improved by adding citation forms not found in the lexicon to the Verb Paradigm Repository (VPR).
  • The current approach focuses on verbs; extending it to other grammatical categories like nouns and adverbs is a natural next step.
  • Future Work:
    • Augmenting the corpus to improve recall.
  • Applying the methodology to other grammatical categories.
  • Further enhancement of the suffix list.
  • Exploring the application of this approach to other morphologically rich languages.

Summary updated

BibTeX
@inproceedings{desai2012automated,
  author = {Shilpa Desai and Jyoti Pawar and Pushpak Bhattacharyya},
  title = {{Automated Paradigm Selection for FSA based Konkani Verb Morphological Analyzer}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2012},
}
Figure