TL;DR
This paper introduces a semi-automatic tool for generating "sortal constraints" – rules that define how different types of words can combine semantically – for natural language understanding systems. By automating much of this knowledge acquisition process, the tool significantly reduces the effort required to adapt systems to new domains, as demonstrated by its application to the ATIS (Air Travel Information System) domain.
What Problem Does This Paper Address?
Building robust natural language understanding (NLU) systems requires extensive domain-specific knowledge. A crucial part of this knowledge is understanding how words and phrases can be combined meaningfully. This is often captured by "sortal constraints," which act as semantic selectional restrictions, ensuring that, for example, a "flight" can "depart" but a "city" cannot. Manually creating these constraints for each new domain is a time-consuming and labor-intensive process, especially when porting an existing system. This paper addresses the challenge of automating and accelerating this knowledge acquisition process.
Methods
The core of the paper is the description of a tool that semi-automatically acquires sortal constraints. The process involves several key steps:
- Signature Generation: An initial "signature file" is created, which assigns inherent sorts to lexical items based on their grammatical category. This is largely automated from the lexicon.
- Corpus Parsing: A large corpus of text from the target domain is parsed using the NLU system (Gemini in this case). The parser generates logical forms that represent the semantic structure of sentences, initially relying only on syntactic constraints.
- Sort Rule Extraction: The tool then recursively explores these logical forms to extract preliminary sort rules. For example, if a parse shows a "flight" acting as the "actor" of a "departure," a sort rule linking these is generated.
- Sort Editing: A crucial interactive component, the "sort editor," allows a linguist to review, filter, and refine the automatically extracted sort rules. This involves statistical information about rule occurrences and the ability to view sample sentences associated with specific rules.
- Iterative Refinement: The process is iterative. The refined sort file is used in subsequent parsing and extraction steps, leading to increasingly accurate and complete sortal constraints.
Datasets
- ATIS Domain Corpus: The primary dataset used for evaluation is the ATIS (Air Travel Information System) domain corpus. The paper specifically mentions using the MADCOW corpus, which contains 7,243 sentences of varying lengths and broad linguistic coverage within the ATIS domain.
- Reference Sorts File: For evaluation, the authors used a pre-existing, hand-built "reference sorts file" for the ATIS domain. This file served as a benchmark against which the automatically generated sorts were compared.
Key Insights and Ideas
- Automation Reduces Bottleneck: The most significant insight is that automating the initial extraction of sortal constraints dramatically reduces the bottleneck in adapting NLU systems to new domains.
- Iterative Refinement is Key: The semi-automatic approach, combining automated extraction with interactive human editing, is essential. The tool provides statistical data and sentence examples to guide the linguist's refinement process.
- Preferred Logical Forms (PLFs) Improve Efficiency: Using Preferred Logical Forms (PLFs) – the most likely interpretations of sentences – rather than all possible logical forms, led to a more efficient extraction of relevant sort rules and a higher proportion of "exact" matches with the reference file.
- Signature File as a Foundation: The signature file, derived from the lexicon, acts as a crucial starting point, providing inherent sorts for lexical items that then help build more complex relational sort rules.
Key Results
- The first iteration of parsing the MADCOW corpus yielded 5,917 sort rules from LFs and 2,275 from PLFs.
- When comparing extracted rules against a reference file, the PLF mode showed a higher percentage of "exact" matches (40% vs. 27%) and "subsumed-by" rules (47% vs. 22%), indicating better alignment with the hand-crafted knowledge.
- The precision of the tool for the PLF mode after extraction was estimated to be between 16% (exact rules) and 55% (exact + subsumed rules).
- The recall for the PLF mode after parsing was estimated to be at least 57%.
- The authors note that the tool significantly reduces the time linguists spend on knowledge acquisition, with precision converging rapidly after a few editing iterations.
Limitations and Future Work
- Manual Effort Still Required: While automated, the process still requires significant manual effort from a linguist for editing and refinement. The paper acknowledges that some parts of creating the signature file (e.g., for domain-specific words) need to be done by hand.
- Evaluation Challenges: Evaluating the precise contribution of sorts versus grammar is inherently difficult. The authors used a pre-existing reference file as a proxy for "correct" sorts.
- Future Work: The authors suggest that a more precise evaluation could be achieved by using a sortal hierarchy to generate all subsumed rules for both corpus and reference sets. They also mention that the tool can be used to improve existing knowledge bases by identifying incompatible rules.
Summary updated