Automatic Construction of Semantic Lexicons for Learning Natural Language Interfaces

AAAI 1999

TL;DR

This paper introduces WOLFIE, a system that automatically learns semantic lexicons (word-to-meaning mappings) from annotated sentences. This lexicon is then used by another system (CHILL) to build natural language interfaces that can parse queries into executable logical forms, demonstrating comparable performance to manually built systems.

What Problem Does This Paper Address?

Building natural language interfaces (NLIs) often requires extensive manual effort, particularly in creating the semantic lexicon – the crucial component that maps words and phrases to their meanings. This lexicon is essential for systems that translate human language into formal representations, like database queries. Manually constructing and maintaining these lexicons is time-consuming and error-prone, especially when dealing with new domains or languages. This paper tackles the challenge of automating this lexicon acquisition process, aiming to simplify the development of NLIs.

Methods

WOLFIE employs a batch, greedy learning algorithm to acquire a semantic lexicon. The core idea is to find (phrase, meaning) pairs that can collectively explain the semantic representations of the training sentences.

Here's a breakdown of the key steps:

  • Candidate Meaning Generation: For each phrase (initially one or two words), WOLFIE identifies potential meanings by finding common substructures (Largest Isomorphic Connected Subgraphs - LICS) within the semantic representations of sentences containing that phrase. This leverages the assumption of compositionality – that a sentence's meaning is built from the meanings of its constituent phrases.
  • Greedy Lexicon Construction: WOLFIE then iteratively selects the "best" (phrase, meaning) pair to add to the lexicon. The "best" is determined by a heuristic that balances the probability of correctly predicting a meaning for a phrase with the generality of the meaning itself. This process aims to cover the training data while minimizing lexicon ambiguity and size.
  • Integration with a Parser: WOLFIE is designed to work in conjunction with CHILL, a system that learns parsers using inductive logic programming. CHILL requires a semantic lexicon to build its parser. By providing WOLFIE's learned lexicon to CHILL, the authors demonstrate an end-to-end system for NLI construction.

Datasets

The primary dataset used is a corpus of 250 questions about U.S. geography, paired with their logical form representations. This corpus was originally used to build a hand-coded NLI called Geobase.

For cross-lingual evaluation, this corpus was translated into:

  • Spanish
  • Japanese (with word segmentation)
  • Turkish

A larger, more diverse geography corpus was also used, containing 471 sentences (250 original + 221 new) collected from computer science students, which included more complex queries and novel words.

Key Insights and Ideas

  • Automating Lexicon Acquisition is Feasible and Effective: The paper demonstrates that a significant portion of the semantic lexicon can be learned automatically, reducing manual effort.
  • Compositionality is Key: The algorithm relies heavily on the principle that sentence meanings are composed from phrase meanings, allowing it to infer potential meanings from shared substructures.
  • Greedy Approach Works Well: Despite the complexity of the problem, a straightforward greedy search strategy, combined with a well-chosen heuristic, yields accurate results.
  • Integrated Systems are Powerful: The success of WOLFIE is amplified by its integration with CHILL, showing that learned components can be effectively combined to build a complete NLI.
  • Cross-Lingual Transfer is Possible: The system's ability to learn lexicons for Spanish, Japanese, and Turkish suggests that the underlying principles are language-independent.

Key Results

  • Comparable Performance to Hand-Built Lexicons: In the U.S. geography domain, parsers built using WOLFIE's learned lexicons achieved accuracy nearly as high as those built with a hand-coded lexicon.
  • Superior to Competing Lexicon Learner: WOLFIE outperformed Siskind's (1996) incremental lexicon learner, achieving higher accuracy and producing smaller, less ambiguous lexicons.
  • Cross-Lingual Success: The system successfully learned lexicons for Spanish, Japanese, and Turkish, with performance differences across languages being minimal.
  • Robustness on a More Complex Corpus: Even with a larger, more diverse corpus containing novel words, WOLFIE's performance remained strong, with no statistically significant difference compared to a system using an extended hand-built lexicon.
  • Lexicon Size and Ambiguity: WOLFIE learned lexicons that were significantly smaller and less ambiguous than those learned by Siskind's system.

Limitations and Future Work

  • Assumptions: The system currently assumes compositionality, that each representation component comes from exactly one phrase, and that a phrase's meaning is a connected subgraph of the representation. Relaxing these assumptions is a direction for future work.
  • Phrase Length: The current implementation is limited to one and two-word phrases, though it can be extended to longer phrases with increased complexity.
  • Search Heuristic: While the greedy search performed well, exploring better search heuristics or alternative strategies could lead to further improvements.
  • Active Learning: The authors suggest using active learning to reduce the burden of annotating training data, allowing the system to select the most informative examples.
  • SQL Domain: Extending the approach to learn parsers for SQL database queries is a promising avenue, as such corpora are more readily available.

Summary updated

BibTeX
@inproceedings{1999automatic,
  title = {{Automatic Construction of Semantic Lexicons for Learning Natural Language Interfaces}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1999},
}
Figure