Dialog Intent Induction with Deep Multi-View Clustering

Hugh Perkins, Yi YangEMNLP 2019 · DOI

TL;DR

This paper introduces "dialog intent induction," the task of automatically discovering user intents from conversations, and proposes a novel deep multi-view clustering approach called AV-KMEANS. AV-KMEANS leverages both user queries and the rest of the conversation to learn better intent representations, outperforming existing methods on real-world customer service and question-answering datasets.

What Problem Does This Paper Address?

Goal-oriented dialog systems are crucial for applications like customer support and virtual assistants. A fundamental step in building these systems is understanding the user's intent. However, existing systems often assume predefined intents, which limits their adaptability to complex domains like customer support where intents can be numerous and nuanced. Manually defining these intents is labor-intensive and doesn't scale well. This paper addresses the challenge of automatically discovering these intents from raw human-human conversations, enabling more robust and adaptable dialog systems.

Methods

The core contribution is Alternating-view k-means (AV-KMEANS), a deep multi-view clustering algorithm. The key idea is to treat a dialog as having two "views": the user's initial query and the remainder of the conversation. AV-KMEANS uses two separate neural network encoders, one for each view, to learn representations. It then iteratively updates these encoders by:

  1. Clustering: Performing k-means clustering on the representations from one view.
  2. Projection & Training: Using the cluster assignments from step 1 as "pseudo-labels" to train the encoder for the other view using a supervised learning approach (specifically, prototypical networks).

This alternating process encourages the representations from both views to yield similar cluster assignments, effectively learning representations that are discriminative for intent clustering. The paper also explores different pretraining strategies for the neural encoders, including autoencoders and the "quick thoughts" method, to improve performance.

Datasets

The authors evaluate their approach on two datasets:

  • Twitter Airline Customer Support (TwACS): A newly curated dataset derived from a large Twitter corpus of customer service conversations in the airline industry. It contains 43,072 dialogs and was manually annotated with 14 dialog intents.
  • AskUbuntu: A dataset originally collected for duplicate question detection, reprocessed to create a question intent clustering dataset. It consists of 1,029 question-answer pairs.

Key Insights and Ideas

  • Leveraging Conversation Context is Crucial: The paper strongly demonstrates that using the entire conversation, not just the user's query, significantly improves dialog intent discovery. Even when queries are lexically dissimilar, the conversational context can reveal the underlying intent.
  • Joint Representation Learning and Clustering is Powerful: AV-KMEANS's ability to jointly learn representations and perform clustering outperforms methods that treat these as separate steps or only learn representations without explicit clustering objectives.
  • Pretraining Matters, Especially with Limited Data: For the smaller TwACS dataset, pretraining the encoders (especially with "quick thoughts") significantly boosts performance. For the larger AskUbuntu dataset, the impact of pretraining is less pronounced, suggesting that sufficient data can compensate for the lack of pretraining.

Key Results

AV-KMEANS consistently outperforms baseline methods, including standard k-means, spectral clustering, and representation learning techniques like autoencoders and quick thoughts.

  • On the TwACS dataset, AV-KMEANS with "quick thoughts" pretraining achieved an F1 score of 46.2% and an ACC of 39.9%, significantly outperforming the best baseline (k-means with quick thoughts) by 4.5% ACC.
  • On the AskUbuntu dataset, AV-KMEANS with "quick thoughts" pretraining achieved an F1 score of 53.3% and an ACC of 41.1%, outperforming the best baseline by a substantial 12.2% F1.
  • The paper also highlights that multi-view spectral clustering (MVSC) performs worse than single-view k-means when using representations from methods like "quick thoughts," suggesting that traditional multi-view methods struggle when representations are already rich and capture cross-view semantics.

Limitations and Future Work

  • Error Analysis: The authors note that their best system still struggles with the "Other" category, which encompasses miscellaneous intents, and sometimes confuses intents that are semantically related (e.g., "FlightDelay" and "TerminalOperation").
  • Multi-Intent Queries: The current approach assumes a single intent per query. The paper acknowledges that users can express multiple intents in a single utterance and suggests this as a direction for future work.
  • Generalization to Multi-Lingual/Modal: The authors propose extending their approach to handle multi-lingual or multi-modal data sources and investigating its effectiveness on a wider range of tasks.

Summary updated

BibTeX
@inproceedings{perkins2019dialog,
  author = {Hugh Perkins and Yi Yang},
  title = {{Dialog Intent Induction with Deep Multi-View Clustering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1413},
}
Figure