Unsupervised Slot Schema Induction for Task-oriented Dialog

Dian Yu, Mingqiu Wang, Yuan Cao, Izhak Shafran, Laurent El Shafey, Hagen Soltau

Abstract

Building task-oriented dialog (TOD) systems traditionally requires extensive manual effort to define schemas—the underlying structure of intents and slots. This paper presents a fully unsupervised pipeline developed by researchers at UC Davis and Google Research to automatically induce slot schemas from raw, unlabeled dialogs. By integrating in-domain language model pre-training with unsupervised grammar structures and hierarchical clustering, the proposed method significantly outperforms existing baselines and demonstrates strong performance in zero-shot dialog state tracking.

Introduction

Task-Oriented Dialog (TOD) systems, such as virtual assistants for booking travel or managing schedules, rely on a predefined "schema." This schema acts as a blueprint, defining specific intents (e.g., book_hotel) and slots (e.g., price_range, check_in_date) that the system must recognize to be functional.

Historically, defining these schemas has been a bottleneck. Experts must manually analyze thousands of conversation logs to identify relevant slots, a process that is expensive, slow, and often biased. While automated methods have been explored, they frequently rely on supervised tools—like pre-trained parsers—that fail to generalize to niche or emerging domains. This research addresses the challenge of Unsupervised Slot Schema Induction, aiming to discover conversational structure without any human-labeled data.

Methodology

The researchers propose a two-stage framework designed to extract and categorize information without supervision.

1. Bottom-up Span Extraction

The first stage involves identifying "informative phrases" or spans (e.g., "after 16:15" or "expensive") within the dialog. The system utilizes a Pre-trained Language Model (LM) to calculate the semantic distance between adjacent tokens based on their attention distributions.

To ensure these spans are linguistically coherent, the authors introduce two key enhancements:

  • PCFG Regularization: The system uses an unsupervised Probabilistic Context-Free Grammar (PCFG). Tokens are only merged into a phrase if they are semantically similar and belong to the same structural branch in a predicted grammar tree, filtering out "fuzzy" or non-grammatical spans.
  • TOD-Span Pre-training: The LM undergoes in-domain pre-training specifically on dialog data, improving its ability to recognize boundaries in conversational speech.

2. Multi-step Auto-tuned Clustering

Once spans are extracted, they must be categorized into meaningful slot types. The framework employs HDBSCAN, a density-based clustering algorithm, in a coarse-to-fine hierarchical approach:

  • Step 1 (General Meaning): Spans are grouped by their surface-level semantics (e.g., grouping all time-related phrases).
  • Step 2 (Domain Separation): The system uses the context of the entire utterance to distinguish between domains (e.g., separating "time" for a taxi booking from "time" for a restaurant reservation).
  • Step 3 (Slot Refinement): These groups are refined into specific slot types (e.g., departure_time vs. arrival_time).

To eliminate the need for manual hyperparameter tuning, the number of clusters is determined automatically by maximizing the Silhouette coefficient, ensuring the resulting schema is data-driven rather than human-guessed.

Experimental Setup

The framework was evaluated on two primary benchmarks:

  • MultiWOZ: A standard multi-domain TOD dataset featuring complex interactions across domains like hotels, trains, and restaurants.
  • SGD (Schema-Guided Dialogue): A large-scale dataset containing over 16,000 conversations across 16+ domains, designed to test generalization to unseen schemas.

Key Results and Insights

The proposed TOD-Span + PCFG configuration demonstrated significant improvements over existing methods:

  • Schema Induction Accuracy: On MultiWOZ, the model achieved a 96.67% F1 score for slot type induction, a substantial increase over the previous DSI baseline of 87.72%.
  • Zero-Shot Dialog State Tracking (DST): When induced clusters were mapped to ground-truth labels, the model achieved a 46.69% Joint Goal Accuracy on MultiWOZ. This result is particularly notable as the model never encountered a single labeled example during training.
  • Contextual Role Recognition: The researchers found that replacing a candidate span with a [MASK] token during clustering helped the model focus on the role of the word within the sentence rather than just its literal meaning.

Limitations and Future Work

Despite its success, the model faces challenges with rare slots (e.g., slots appearing only once in the dataset) and boundary precision (e.g., extracting "free wifi" when the ground truth is a binary "yes"). Additionally, the current system does not account for complex linguistic features like coreference or ellipses.

Future research will focus on extending this unsupervised framework to handle more complex data structures and applying the methodology to related tasks such as unsupervised Question Answering.

Summary updated

BibTeX
@inproceedings{yu2022unsupervised,
  author = {Dian Yu and Mingqiu Wang and Yuan Cao and Izhak Shafran and Laurent El Shafey and Hagen Soltau},
  title = {{Unsupervised Slot Schema Induction for Task-oriented Dialog}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=4SlVHAAGmY-},
}
Figure