Knowledge-Driven Slot Constraints for Goal-Oriented Dialogue Systems

Piyawat Lertvittayakumjorn, Daniele Bonadiman, Saab MansourNAACL 2021 · DOI

Introduction

Goal-oriented dialogue systems, such as those used for food ordering or insurance inquiries, rely on Natural Language Understanding (NLU) to extract user intents and slot values. However, a significant challenge arises when user requests conflict with real-world business logic—for instance, ordering a "large" size for a menu item that only exists in "small." Traditionally, these knowledge-driven constraints are handled as a post-processing step, which often leads to error accumulation and a lack of transparency for the user.

This paper introduces a formal framework for constraint violation detection directly within the NLU unit. By identifying invalid slot combinations early, systems can provide more accurate feedback and handle "unlinkable" entities (mentions that do not match the database) more gracefully.

Methodology

The authors formalize constraints using atomic formulas and logical operators. The following example illustrates how these constraints are structured logically against the NLU output:

Example of an input utterance and its corresponding formal logical constraint.

To detect violations, the researchers evaluated three distinct architectural approaches:

Comparison of the Deterministic Pipeline, Probabilistic Pipeline, and End-to-End architectures.
  1. Deterministic Pipeline (DP): A sequential three-step process involving Intent Classification and Slot Labeling (IC/SL), Entity Linking (mapping text to database entities), and a hard-coded logic check.
  2. Probabilistic Pipeline (PP): Similar to the DP, but maintains a probability distribution over possible entities to calculate a violation score based on the likelihood of all valid combinations.
  3. End-to-End (EE): A BERT-based model that treats violation detection as a multi-label classification task, predicting violations directly from raw text and bypassing explicit slot filling.

Experimental Setup

The framework was evaluated using the MultiDoGO dataset across two domains of varying complexity:

  • Insurance: A relatively simple domain with 3 custom slot types and 4 constraints (primarily "closed type" checks).
  • Fast Food: A complex domain featuring 6 custom slot types and 18 constraints, such as validating specific toppings for specific pizza types.

The authors manually augmented these datasets with constraint definitions and turn-level violation labels, releasing the benchmark to the research community.

Key Results and Insights

Performance varied significantly based on domain complexity. In the Insurance domain, the End-to-End (EE) approach was the clear winner, achieving 92.1% turn accuracy by learning simple patterns directly from the data. However, in the Fast Food domain, the Deterministic Pipeline (DP) performed best, though accuracy dropped to 50.2%. This highlights the difficulty deep learning models face when learning complex, multi-slot logical dependencies without an explicit schema.

The Entity Linking Bottleneck

Pipeline approaches are heavily dependent on the quality of the entity linker. If a mention like "bbq chicken" cannot be mapped to "Original BBQ Chicken Pizza," a violation is triggered regardless of the user's actual intent. To mitigate this, the authors employed Natural Language Inference (NLI), using a pre-trained RoBERTa model to determine if a slot mention entails a database entity. This proved more robust than standard string matching for varied natural language.

Conclusion and Future Work

While End-to-End models show promise for simple domains, they require substantial data to learn complex logic. Conversely, pipeline approaches suffer from cascading errors. Future research directions include joint learning, where slot filling and violation detection are learned simultaneously, allowing logical constraints to act as a regularizer for the NLU. Additionally, translating logical constraints into synthetic training data could help EE models learn complex business rules more efficiently.

Summary updated

BibTeX
@inproceedings{lertvittayakumjorn2021knowledge,
  author = {Piyawat Lertvittayakumjorn and Daniele Bonadiman and Saab Mansour},
  title = {{Knowledge-Driven Slot Constraints for Goal-Oriented Dialogue Systems}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.266},
}
Figure