Intent Discovery for Enterprise Virtual Assistants: Applications of Utterance Embedding and Clustering to Intent Mining

Minhua Chen, Badrinath Jayakumar, Michael Johnston, S. Eman Mahmoodi, Daniel PresselNAACL 2022 · OpenReview

Abstract

Minhua Chen, Badrinath Jayakumar, Michael Johnston, S. Eman Mahmoodi, Daniel Pressel. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track.

2022.

Introduction

Building effective enterprise virtual assistants—such as customer service bots for insurance or utilities—begins with "Intent Discovery." This process involves identifying what customers are requesting from vast quantities of unlabeled chat logs or call transcripts. While generic language models like BERT excel at general tasks, they often falter in enterprise settings due to domain mismatch, semantic shifts (where contextually similar phrases like "my screen is broken" and "power button not responding" share an intent but differ linguistically), and the "linear trap" of standard softmax loss, which does not optimize for the geometric proximity required for clustering.

Methodology: The Three-Step Pipeline

The authors propose a specialized pipeline designed to create clustering-friendly utterance embeddings.

The three-step pipeline: Generic Transformer Pre-training, Domain-adaptive Transformer Pre-training, and Downstream Embedding and Clustering.
  1. Generic Transformer Pre-training: An 8-layer Transformer is trained on a large public corpus (Reddit, Wikipedia, etc.) using a Masked Language Modeling (MLM) objective.
  2. Domain-adaptive Pre-training: The model is further trained on a "data lake" of 20 enterprise applications using a multi-task cosine softmax loss. This multi-task approach shares an encoder across applications while using distinct "heads" for specific intents. The cosine softmax loss minimizes the angular distance between utterances and intent centroids, preserving the geometric properties essential for clustering.
  3. Downstream Clustering: For a new, unseen application, the trained encoder generates embeddings which are then grouped using K-means clustering.

Experimental Setup

The pipeline was evaluated using:

  • Training Data: Three years of Reddit data and online forums for Step 1; an internal "data lake" of 20 enterprise applications (e.g., retail, travel) for Step 2.
  • Evaluation Data: Four unseen applications—Collections, Power Utility, Insurance, and Photography—to test generalization across business domains.

Key Results and Insights

The proposed approach (Step 1 + 2 + 3) significantly outperforms state-of-the-art baselines like Sentence-BERT (SBERT) and Deep Embedded Clustering (DEC).

Clustering accuracy (ACC) vs. number of clusters (K) for four unseen applications, comparing the proposed pipeline against various baselines.
  • Performance Gains: The full pipeline achieved up to a 20% absolute improvement in clustering accuracy (ACC) over SBERT across various numbers of clusters (K).
  • Criticality of Domain Adaptation: Models skipping Step 2 performed significantly worse, highlighting that generic models lack the nuance required for specialized enterprise language.
  • Geometric Optimization: By using cosine softmax, the model effectively "pre-clusters" data during training, making it easier for K-means to identify meaningful groups in new datasets.
  • Robustness: The model successfully handles the noise and brevity typical of real-world enterprise data, including speech-to-text errors.

Conclusion and Future Work

This research demonstrates that domain-adaptive pre-training with a geometry-preserving loss function is vital for intent discovery in enterprise environments. While successful, the authors note that future work should address privacy risks associated with effective vector searches and explore alternative losses like Supervised Contrastive Learning. Additionally, finding more computationally efficient alternatives to spectral clustering remains a priority for scaling to even larger datasets.

Summary updated

BibTeX
@inproceedings{chen2022intent,
  author = {Minhua Chen and Badrinath Jayakumar and Michael Johnston and S. Eman Mahmoodi and Daniel Pressel},
  title = {{Intent Discovery for Enterprise Virtual Assistants: Applications of Utterance Embedding and Clustering to Intent Mining}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=uLOVr2mdt7},
}
Figure