Discovering Dialogue Slots with Weak Supervision

Vojtech Hudecek, Ondrej Dusek, Zhou YuACL 2021 · DOI

TL;DR

This paper introduces a novel method for automatically discovering and labeling "slots" (key pieces of information) in task-oriented dialogue systems, eliminating the need for expensive manual annotation. By leveraging existing linguistic tools and unsupervised clustering, their approach significantly improves the performance of dialogue systems, particularly in downstream tasks like response generation.

What Problem Does This Paper Address?

Task-oriented dialogue systems, like those used for booking flights or making restaurant reservations, rely on understanding user requests by extracting specific pieces of information called "slots" (e.g., "price," "food type," "area"). Traditionally, training these systems requires a large amount of manually annotated data, where human annotators label these slots in user utterances. This manual annotation process is time-consuming and costly, creating a bottleneck for developing and deploying such systems. While some approaches try to reduce this burden, they often still require significant manual effort or rely on implicit, less interpretable dialogue state representations. This paper aims to completely remove the need for manual slot annotation by discovering relevant slots and training a tagger using only weakly supervised signals.

Methods

The proposed method consists of three main stages:

  1. Weak Supervision Label Acquisition: The system starts with an unlabeled corpus of in-domain dialogues. It then uses off-the-shelf, domain-agnostic linguistic annotation tools, such as frame-semantic parsers (like FrameNet) and Named Entity Recognition (NER) models, to generate initial "slot candidates." These tools provide a form of "weak supervision" because their annotations are not specifically tailored to the target domain.
  2. Domain-Relevant Slot Discovery (Merging and Ranking): This is the core of the novelty.
  3. Candidate Merging: Similar slot candidates identified by different tools (e.g., "Direction" and "Location" both relating to "area") are merged into a single, more robust slot. This is done by measuring similarity based on cosine similarity of embeddings and contextual co-occurrence.
  4. Candidate Ranking and Selection: The system then ranks the merged candidates to identify the most domain-relevant ones. This involves clustering verb-slot pairs to understand contexts, and then ranking candidates based on frequency, coherence (how similar the fillers are), and a TextRank-based score. A thresholding mechanism is used to select the most promising slots.
  5. Slot Tagger Training: Finally, the discovered and filtered set of domain-relevant slots is used to train a standalone neural slot tagger. This tagger is trained only on the outputs of the previous stages, meaning it requires no human-labeled data. This tagger can then be used to annotate new dialogue data efficiently and accurately.

Datasets

The authors evaluate their method on several publicly available datasets:

  • CamRest676 (CR): A restaurant recommendation domain dataset.
  • MultiWOZ (WH and WA): Two domains (hotel reservation and attraction recommendation) from a larger multi-domain corpus.
  • Cambridge SLU (CS): Another restaurant domain dataset.
  • ATIS (AT): A flight information domain dataset.

They use these datasets for both training the slot tagger and evaluating the discovered slots and the downstream dialogue generation task.

Key Insights and Ideas

  • Eliminating Manual Annotation is Feasible: The most significant insight is that it's possible to achieve competitive slot tagging performance without any manual annotation, by cleverly combining existing linguistic tools and unsupervised learning.
  • Weak Supervision + Clustering = Powerful Slot Discovery: The combination of initial, noisy annotations from generic tools with unsupervised clustering and ranking is highly effective at filtering out irrelevant information and identifying domain-specific slots.
  • Standalone Tagger is Crucial for Recall: Training a dedicated slot tagger after discovering the slots is essential for improving recall, as it allows the model to learn domain-specific patterns that the initial weak supervision tools might miss.
  • Domain Independence: The method demonstrates robustness across multiple domains, suggesting it can be applied broadly without extensive domain adaptation.

Key Results

  • State-of-the-Art Weakly Supervised Slot Tagging: The "Ours-full" method (combining slot discovery and the trained tagger) achieves state-of-the-art performance in weakly supervised slot tagging, showing a 6-16% absolute F1 score increase over previous benchmarks on four different domains.
  • Improved Dialogue Response Generation: When the discovered slots are used to train an end-to-end dialogue response generation system (Sequicity), performance significantly improves. For instance, it leads to a 5% increase in correctly identifying the venue and a 35% increase when no prior restaurant ontology is provided.
  • Tagger Boosts Performance: The trained slot tagger, even when trained on only 10% of the original weakly supervised data, significantly improves both precision and recall compared to using the raw output of the weak supervision tools.

Limitations and Future Work

  • Reliance on Linguistic Annotators: The method's performance is inherently limited by the quality and structure of the input from existing linguistic annotation models.
  • Candidate Merging Errors: While necessary, the candidate merging process can sometimes lead to errors where semantically unrelated slots are merged, or rare, irrelevant slots are incorrectly ranked.
  • Future Directions: The authors plan to further improve the model by exploring unsupervised selection of slot candidates via keyword extraction and clustering, and by incorporating contextual information from preceding dialogue turns. They also aim to focus more on the intent detection aspect of dialogue understanding.

Summary updated

BibTeX
@inproceedings{hudecek2021discovering,
  author = {Vojtech Hudecek and Ondrej Dusek and Zhou Yu},
  title = {{Discovering Dialogue Slots with Weak Supervision}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.189},
}

Loading related content...

Figure