Unsupervised Slot Schema Induction for Task-oriented Dialog

Dian Yu, Mingqiu Wang, Yuan Cao, Izhak Shafran, Laurent El Shafey, Hagen Soltau

TL;DR

Building task-oriented chatbots usually requires experts to manually define "schemas"—the specific slots like departure_time or hotel_price a bot needs to track. This research introduces a fully unsupervised method to induce these schemas directly from raw conversation logs, outperforming existing baselines by leveraging in-domain language models regularized by unsupervised grammar structures.

Introduction: The Schema Bottleneck

In Task-Oriented Dialog (TOD) systems, the "schema" (or ontology) serves as the blueprint of the conversation. It defines the specific information the system must extract from a user to complete a task, such as booking a train or a restaurant. Currently, creating these schemas is a significant bottleneck: it is labor-intensive, expensive, and slow. Experts must manually analyze thousands of logs to identify relevant concepts and group them into categories.

While some automated methods exist, they often rely on supervised tools like POS-taggers or predefined semantic frames, which do not always generalize well to niche domains. This paper aims to remove the "human-in-the-loop" requirement entirely, proposing a system that can discover the underlying structure of unlabeled chat logs autonomously.

Methodology

The proposed approach utilizes a two-stage unsupervised pipeline consisting of candidate span extraction followed by multi-step clustering.

Overview of the unsupervised slot schema induction pipeline, showing span extraction and multi-step clustering from dialog logs.

1. Candidate Span Extraction

Instead of relying on standard noun phrase extraction, the authors employ a bottom-up attention-based approach. They measure the "distance" between tokens based on their attention distributions within a pre-trained Language Model (LM). To mitigate noise, this process is regularized using an unsupervised Probabilistic Context-Free Grammar (PCFG). Tokens are merged into a phrase only if they share the same parent in the PCFG tree. The authors further refine this by "in-domain pre-training," creating a model called TOD-Span, specifically trained on dialog data to recognize span boundaries.

2. Multi-step Clustering

Once spans (e.g., "after 16:15" or "cheap") are extracted, they are categorized through a hierarchical process:

  • Step 1: Cluster spans based on contextual embeddings to group general concepts (e.g., time-related phrases).
  • Step 2: Refine clusters using utterance-level representations (the [CLS] token) to distinguish between domains, such as "time" in a train booking vs. a taxi booking.
  • Step 3: Perform fine-grained clustering to isolate specific slot types, such as arrival_time vs. departure_time.

The system uses HDBSCAN for clustering, which is "auto-tuned" by maximizing the Silhouette coefficient, eliminating the need for manual cluster count estimation.

Key Insights

  • Grammar as a Structural Filter: While raw Language Models excel at semantics, they are often imprecise regarding boundaries. Integrating an unsupervised PCFG acts as a structural constraint, significantly reducing noise during span extraction.
  • Hierarchical Contextualization: The multi-step clustering logic mimics human organization: first identifying a general concept, then the intent, and finally the specific functional role.
  • The Power of In-Domain Pre-training: The "TOD-Span" model, pre-trained on task-oriented data with a span-boundary objective, significantly outperformed general-purpose models like standard BERT.
  • Context-Driven Slot Definition: During clustering, candidate spans are replaced with a [MASK] token. This forces the model to define the slot based on the surrounding context rather than just the surface-level text of the value.

Experimental Results

The researchers evaluated their method on two gold-standard benchmarks: MultiWOZ (a multi-domain dataset) and SGD (Schema-Guided Dialog, designed for unseen schemas).

Performance comparison of various methods on MultiWOZ schema induction, highlighting the superiority of the TOD-Span model constrained by PCFG.

As shown in the results table, the best-performing model (TOD-Span + PCFG) achieved a 96.67 F1 score for slot type induction and 58.71 for slot value induction on MultiWOZ, significantly outperforming the DSI baseline. Furthermore, when the induced schema was applied to a zero-shot Dialog State Tracking (DST) task, it achieved a 46.69 joint accuracy, nearly doubling the performance of the baseline.

Limitations and Future Work

Despite its success, the method has several limitations. First, while the system groups values like "cheap" and "expensive" effectively, a human is still required to provide the final label (e.g., price_range) for the cluster. Second, the model struggles with single-value slots (e.g., a "has_wifi" slot that is always "True"), as there is insufficient variety for the clustering algorithm to detect a pattern. Finally, the current model does not handle complex linguistic features like coreferences or ellipses. Future research may extend this framework to handle more complex, hierarchical slot dependencies.

Summary updated

BibTeX
@inproceedings{yu2022unsupervised,
  author = {Dian Yu and Mingqiu Wang and Yuan Cao and Izhak Shafran and Laurent El Shafey and Hagen Soltau},
  title = {{Unsupervised Slot Schema Induction for Task-oriented Dialog}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=4SlVHAAGmY-},
}
Figure