TL;DR
This paper presents a system for automatically generating computational grammars for natural languages based on linguistic questionnaire responses. By applying this system to Sahaptin, a language with complex argument marking, the authors demonstrate its effectiveness, identify areas for improvement in the system, and uncover gaps in existing linguistic descriptions of Sahaptin.
What Problem Does This Paper Address?
Building computational grammars for natural languages is a labor-intensive process. The Grammar Matrix project aims to streamline this by providing a framework and a customization system. However, many linguistic phenomena are widespread but not universal, posing a challenge for systems that only support universal features. The authors address this by developing a web-based questionnaire that elicits language-specific information to automatically generate a tailored grammar. This paper specifically tests this customization system with Sahaptin, a language chosen for its intricate argument marking and agreement patterns, to evaluate the system's capabilities and identify potential shortcomings.
Methods
The core of the paper revolves around the Matrix customization system. This system presents linguists with a web-based questionnaire designed to capture the grammatical features of a target language. Based on the answers provided, the system automatically generates a HPSG (Head-driven Phrase Structure Grammar) implemented in the Type Description Language (TDL) and interpretable by the Linguistic Knowledge Building (LKB) system.
Key aspects of the method include:
- Library-based approach: The system is built with libraries, each supporting a specific linguistic phenomenon (e.g., case marking, agreement, direct-inverse marking).
- Regression testing: To ensure the implementation of new libraries doesn't break existing ones, a regression test suite is used. This involves providing a set of answers to the questionnaire, a list of grammatical and ungrammatical sentences, and the expected parsing results.
- Sahaptin as a test case: The authors applied the customization system to Sahaptin, a language with complex argument marking and agreement. This involved analyzing linguistic descriptions of Sahaptin and translating that analysis into answers for the questionnaire.
- Test sentence generation: A comprehensive set of test sentences, both grammatical and ungrammatical, was created to thoroughly probe the generated Sahaptin grammar. This included permutations of known grammatical sentences and programmatically generated sentences to cover a wide range of linguistic phenomena.
Datasets
The paper does not explicitly mention pre-existing datasets for training or evaluation in the traditional machine learning sense. Instead, the "dataset" is effectively the linguistic knowledge of Sahaptin, as described in Rigsby and Rude (1996), which is encoded into the customization system via the questionnaire. The paper also introduces a new benchmark: the Sahaptin test case, comprising 6165 unique test sentences (89 grammatical and 5856 ungrammatical) designed to evaluate the generated grammar.
Key Insights and Ideas
- System's Generality: The customization system, even without being specifically designed for Sahaptin, proved capable of modeling its complex argument marking and agreement patterns. This highlights the system's flexibility and broad typological coverage.
- System-Driven Linguistic Discovery: The process of creating the Sahaptin grammar revealed gaps and ambiguities in existing linguistic descriptions of the language. This demonstrates how computational grammar development can actively contribute to linguistic research.
- Need for Scale-Sensitive Agreement: The Sahaptin case motivated the addition of a new library for scale-sensitive agreement to the customization system. This type of agreement, where morphemes depend on the relative ranking of arguments on a scale (like direct-inverse), is crucial for languages like Sahaptin and Plains Cree.
- Trade-offs in Homophony and Hierarchy: The authors navigated the complexity of Sahaptin morphology by balancing homophony (multiple meanings for the same form) and hierarchical complexity in feature structures, aiming for a balance between understandability and descriptive accuracy.
Key Results
- The generated Sahaptin grammar correctly analyzed all 89 grammatical test sentences.
- Out of 5856 ungrammatical sentences, 5848 failed to parse as expected.
- A surprising eight ungrammatical sentences were parsed, indicating potential areas for refinement in the grammar or the test set.
- The Sahaptin test case, developed using the system, now serves as a regression test for the customization system and a sample grammar for users.
- The project led to the identification of previously unnoticed gaps in Sahaptin linguistic descriptions, which were then addressed in collaboration with linguists.
Limitations and Future Work
- Morphophonology: The customization system models complex morphosyntax but not complex morphophonology. It expects morpheme-by-morpheme glosses as input, leaving morphological analysis to other systems.
- Word Order: The system's current word order library cannot model truly free word order or discontinuous noun phrases. The Sahaptin grammar was simplified to a VSO order with enclitics as suffixes on verbs.
- Simplifying Assumptions: Due to system limitations, some simplifying assumptions were made regarding Sahaptin's morphosyntax.
- Future Work: The authors suggest enhancing the word order library to support second-position clitics. The Sahaptin test case itself is now integrated into the system as a regression test and a user example, facilitating further development and adoption. The paper also highlights the ongoing feedback loop between grammar engineering and system development, where new language analyses inform system improvements.
Summary updated