Discovering New Intents with Deep Aligned Clustering

Hanlei Zhang, Hua Xu, Ting-En Lin, Rui Lv

Introduction

In modern dialogue systems, user intents are dynamic and ever-evolving. While systems are typically trained on a set of "known intents" (e.g., check_balance), new categories frequently emerge. Manually labeling these new intents is labor-intensive and unscalable. Deep Aligned Clustering (DAC) addresses this by leveraging limited labeled data from known intents to discover and categorize novel ones.

The Challenge of Intent Discovery

The task of discovering new intents presents two primary hurdles:

  1. Knowledge Transfer: Effectively utilizing a small amount of labeled data to guide the discovery of unseen categories.
  2. Label Inconsistency (The "Moving Target" Problem): Iterative deep clustering often suffers from shifting cluster assignments. A group identified as "Cluster 1" in one epoch might be labeled "Cluster 5" in the next, creating an unstable supervision signal that hinders feature learning.

The Deep Aligned Clustering (DAC) Framework

The authors propose a three-stage pipeline to overcome these challenges:

  1. Intent Representation & Pre-training: A BERT-based model is used to extract sentence features. It is initially "warmed up" on labeled data from known intents to establish a baseline understanding of intent structures.
  2. Automated Cluster-Number Estimation: To handle the unknown number of new intents ($K$), the framework starts with an over-estimation and performs K-means. It then prunes "low-confidence" clusters with few samples to find the natural number of categories.
  3. Deep Aligned Clustering: This core component solves the moving target problem. By using the Hungarian Algorithm, the model aligns cluster centroids between the current and previous epochs based on Euclidean distance. This ensures that pseudo-labels remain consistent over time, allowing the network to build upon previously learned knowledge.
Overview of the Deep Aligned Clustering (DAC) framework, highlighting the centroid alignment strategy.

Experimental Setup

DAC was evaluated on two high-quality datasets:

  • CLINC: 150 intents across 10 domains (e.g., travel, banking).
  • BANKING: 77 fine-grained intents specifically focused on the banking domain.

Experiments simulated a real-world scenario where 75% of intents were "known" and 25% were "new," with only 10% of the known intent data being labeled.

Key Results

DAC significantly outperforms existing state-of-the-art methods like CDAC+ and DeepCluster. On the CLINC dataset, DAC achieved an accuracy of 86.49%, a substantial improvement over previous semi-supervised methods that typically reached 70-74%.

Performance comparison of DAC against baselines on the CLINC dataset across NMI, ARI, and ACC metrics.

The framework also demonstrated remarkable robustness. As shown in the results, DAC maintains high performance even when the ratio of known classes is low (e.g., 25%), whereas competing models often experience sharp performance declines. Ablation studies further confirmed that the alignment strategy alone provides a 10-30% boost in accuracy compared to standard reinitialization methods.

Discussion and Future Work

The success of DAC highlights that the consistency of the supervision signal is as critical as the clustering algorithm itself. By using stable centroids as anchors for alignment, the authors transformed a chaotic unsupervised process into a stable self-supervised one.

However, some limitations remain:

  • Threshold Sensitivity: The cluster pruning process depends on a threshold ($t$) that may require tuning for imbalanced datasets.
  • Batch Processing: The current model operates in a batch setting. Future research could adapt DAC into an incremental learning framework to discover intents dynamically in production environments.
  • Feature Enhancement: Exploring additional self-supervised signals could further improve the discriminative power of intent representations.

Summary updated

BibTeX
@inproceedings{zhang2021discovering,
  author = {Hanlei Zhang and Hua Xu and Ting-En Lin and Rui Lv},
  title = {{Discovering New Intents with Deep Aligned Clustering}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i16.17689},
}
Figure