Identifying Domain Adjacent Instances for Semantic Parsers

James Ferguson, Janara Christensen, Edward Li, Edgar GonzàlezEMNLP 2018 · DOI

Abstract

Semantic parsers often fail silently when users provide inputs that are topically relevant but functionally unsupported. This paper formalizes the problem of domain-adjacent (DA) instance identification and introduces a novel "SURPRISE" weighting method. By highlighting words that are unexpected given their context, the proposed method effectively flags these tricky inputs, allowing systems to gracefully handle queries that fall just outside their capabilities.

The Challenge of Domain Adjacency

Semantic parsing translates natural language into machine-readable logical forms. While research typically focuses on "In-Domain" (supported) and "Out-of-Domain" (completely unrelated) queries, a frustrating middle ground exists: Domain-Adjacent instances.

These occur when a user stays on-topic but requests a feature the system does not support. For example, an air travel bot capable of booking flights might struggle with a request to "change seats" if that specific functionality wasn't implemented. Because the vocabulary is so similar to the training data, standard parsers often attempt to "force" a fit, leading to incorrect or even harmful actions.

Figure 1: Comparison of In-Domain, Domain-Adjacent, and Out-of-Domain predicates in an Air Travel context.

The SURPRISE Weighting Method

The authors propose a two-part approach to identify these "near-miss" queries using a novel sentence representation and a simple classifier.

  1. SURPRISE Sentence Representation: The core insight is that DA sentences contain specific words that are "surprising" in their context. The model learns a mapping from general pre-trained word embeddings to domain-specific embeddings. It then calculates a "surprise weight" for each word by comparing the actual embedding to an "expected" embedding (the average of its neighbors). Unsupported words like "seat" or "miles" receive high weights, ensuring they dominate the final sentence vector.
  2. The Classifier: Interestingly, the researchers found that a simple k-Nearest Neighbors (kNN) approach worked best. By calculating the average cosine distance between a new sentence’s embedding and its $k$ nearest neighbors in the training set, the system can flag queries that exceed a specific distance threshold as domain-adjacent.

Experimental Setup and Results

The researchers utilized the OVERNIGHT dataset, covering eight domains such as Basketball, Recipes, and Housing. To simulate domain adjacency, they employed a "leave-one-predicate-out" experiment, removing specific predicates (e.g., starRating in Restaurants) from the training set to serve as DA test cases.

Key Findings

  • Performance: The SURPRISE model achieved high Area Under the Curve (AUC) scores, particularly in the "Publications" (0.978) and "Restaurants" (0.941) domains.
  • Simplicity vs. Complexity: A major takeaway was that simpler models like kNN and SURPRISE often outperformed complex neural architectures like LSTMs or Autoencoders, which tended to overfit the training data and lose sensitivity to subtle linguistic "surprises."
Figure 4: AUC performance comparison across eight domains, showing the SURPRISE method's competitive performance against neural baselines.

Discussion and Limitations

While highly effective, the model struggled in the "Social" domain. In this dataset, supported and unsupported queries used nearly identical wording (e.g., "start date" vs "end date"), making it difficult to isolate unique "surprising" words.

Future work will focus on recovery strategies—how a system should respond once a DA instance is identified—and methods for setting optimal distance thresholds to balance false positives and negatives in production environments. By correctly blocking DA instances, semantic parsers can avoid incorrect actions and improve overall reliability.

Summary updated

BibTeX
@inproceedings{ferguson2018identifying,
  author = {James Ferguson and Janara Christensen and Edward Li and Edgar Gonzàlez},
  title = {{Identifying Domain Adjacent Instances for Semantic Parsers}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1539},
}
Figure