Open-Vocabulary Semantic Parsing with both Distributional Statistics and Formal Knowledge

Matt Gardner, Jayant KrishnamurthyAAAI 2017 · DOI

Introduction

Semantic parsing—the translation of natural language into machine-readable queries—has traditionally faced a trade-off between precision and coverage. Traditional parsers are highly accurate within a fixed schema (e.g., Freebase) but fail on out-of-vocabulary terms. Conversely, open-vocabulary parsers use large text corpora for broad coverage but lack the structured "ground truth" of a formal Knowledge Base (KB). This paper introduces a hybrid approach that bridges this gap by treating formal KB queries as features for a distributional model.

Methodology

The authors propose a system that combines Combinatory Categorial Grammar (CCG) parsing with structured knowledge features. The parser first generates "surface" logical forms based on the input text.

Overview of the hybrid semantic parsing model combining distributional parameters and formal knowledge base features.

Subgraph Feature Extraction (SFE)

To integrate the KB, the model uses Subgraph Feature Extraction (SFE). This technique identifies paths in the Freebase graph (e.g., DESIGNED → LOCATED_IN) and converts them into binary features. By treating the KB as a source of features rather than a rigid destination, the model remains robust; it leverages structured data when available and falls back on distributional statistics otherwise.

Subgraph Feature Extraction (SFE) illustrating how paths in a knowledge base are converted into features for the model.

The Combined Model

The probability that an entity fits a predicate is calculated using two distinct components: distributional vectors learned from text co-occurrence and weighted KB features extracted via SFE. To manage dimensionality, Pointwise Mutual Information (PMI) is used to select the top 100 most relevant features per predicate.

Results

The model was evaluated on a fill-in-the-blank task using the ClueWeb09 corpus and Freebase. The results demonstrate that structured knowledge provides a massive boost to open-vocabulary systems. The combined model achieved a Mean Average Precision (MAP) of 0.370, representing a 127% relative improvement over the distributional-only baseline of 0.163.

Key insights include:

  • Connectivity: The KB provides critical data for rare entities that lack sufficient text mentions.
  • Compositionality: Unlike many neural approaches, this model maintains a compositional structure, allowing it to understand how sentence components combine into complex queries.

Discussion and Limitations

Despite its success, the model has limitations. The current edge-sequence features cannot always capture fine-grained nuances, such as specific player positions in sports. Additionally, the system still struggles with "zero-mapping" predicates that have no correlate in the KB. Future work may focus on more expressive path features and improved candidate entity generation.

Summary updated

BibTeX
@inproceedings{gardner2017open,
  author = {Matt Gardner and Jayant Krishnamurthy},
  title = {{Open-Vocabulary Semantic Parsing with both Distributional Statistics and Formal Knowledge}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.1609/aaai.v31i1.10980},
}
Figure