Probabilistic Models for Learning a Semantic Parser Lexicon

Jayant KrishnamurthyNAACL 2016

TL;DR

This paper introduces novel probabilistic models for learning semantic parser lexicons directly from question-answer pairs, bypassing the need for manually annotated logical forms or templates. Their approach achieves significant improvements in semantic parser accuracy and efficiency, while also producing more compact lexicons.

What Problem Does This Paper Address?

Semantic parsing aims to translate natural language into formal meaning representations, often for tasks like question answering over databases. Training a semantic parser requires a lexicon, which maps words to their corresponding meaning representation components. However, learning this lexicon is a significant challenge. Existing methods often rely on heuristics, require extensive human input (like lexicon templates or fully annotated logical forms), and may lack guarantees of convergence to an optimal solution. This paper addresses the problem of learning a high-quality semantic parser lexicon efficiently and with minimal human supervision.

Methods

The core of the paper's contribution lies in its novel probabilistic models for lexicon learning. They frame lexicon learning as a problem of learning the parameters of a probabilistic context-free grammar (PCFG) that represents the space of correct logical form derivations for a given question and its associated label (e.g., a set of possible logical forms).

Key aspects of their approach include:

  • Logical Form Derivation Grammar: They construct a PCFG for each training example that captures how logical forms can be derived and aligned with words in the question. This grammar implicitly defines potential lexicon entries.
  • Generative Models: They propose three generative models:
  • Independent Model: Assumes each word in a question is generated independently from a parse tree. This model has a concave objective function, guaranteeing convergence to a global optimum via Expectation Maximization (EM).
  • Coupled Model: Generates the entire question from a single parse tree, removing the independence assumption of the previous model.
  • Coupled Loglinear Model (PAL): Replaces conditional probability tables with loglinear models, allowing for parameter sharing and more sophisticated feature representations. This is the model they primarily evaluate.
  • EM Training: All models are trained using Expectation Maximization (EM) to learn the parameters of the grammar.
  • Lexicon Production: After training, lexicon entries are derived from the most probable parse of each example, mapping words to syntactic categories and logical forms.

Datasets

  • FOODCHAINS: A newly introduced dataset containing 774 food chain questions designed to mimic 4th-grade science exam questions. This dataset features challenging lexical diversity and question length, with logical forms annotated by the authors.
  • GEO880: An existing dataset of geography questions, used to evaluate the models on more complex logical forms.

Key Insights and Ideas

  • Lexicon Learning as Grammar Induction: Framing lexicon learning as learning the parameters of a PCFG that generates logical form derivations is a powerful abstraction.
  • Concave Objective Guarantees: The independent model's concave objective function is a significant advantage, ensuring convergence to a global optimum, which is rare in complex machine learning tasks.
  • Generative vs. Discriminative: Their generative approach, unlike many discriminative baselines, naturally handles the ambiguity of weak supervision and avoids issues with overly specific lexicon entries.
  • Efficiency Gains: The probabilistic models lead to significantly smaller lexicons, which in turn results in much faster semantic parsers.
  • Human Input Reduction: The models achieve competitive results with substantially less human input (e.g., no lexicon templates or annotated logical forms required during training).

Key Results

  • FOODCHAINS Performance:
    • Improved semantic parser accuracy by 35-70% error reduction compared to prior work.
  • Achieved a semantic parser that is 4x faster than the best baseline (POS).
  • Produced lexicons that are up to 40x smaller than the most accurate baseline.
  • GEO880 Performance: Obtained competitive results on GEO880, demonstrating applicability to more complex logical forms, especially when augmented with factored lexicon entries.
  • Model Comparison: The three proposed probabilistic models performed similarly, with the coupled loglinear model (PAL) slightly outperforming the others.

Limitations and Future Work

  • Logical Form Enumeration: The authors acknowledge that enumerating all logical forms that produce a correct answer is computationally intensive. However, they argue this is unavoidable in the question-answering setting and that their approach offers a more principled way to manage this enumeration.
  • Future Work: They plan to apply their approach to construct semantic parsers for more challenging tasks and explore how semantic parser parameter learning might affect logical form enumeration. They also suggest that combining their learned lexicons with manually specified parsing operations could be a promising direction.

Summary updated

BibTeX
@inproceedings{krishnamurthy2016probabilistic,
  author = {Jayant Krishnamurthy},
  title = {{Probabilistic Models for Learning a Semantic Parser Lexicon}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2016},
}
Figure