Constrained Self-Supervised Clustering for Discovering New Intents (Student Abstract)

Ting-En Lin, Hua Xu, Hanlei Zhang

Abstract

Identifying novel user intents is a critical challenge for evolving dialogue systems. This work introduces a self-supervised clustering framework that utilizes a limited set of labeled data to learn intent representations. By treating clustering as a pairwise classification task and employing dynamic thresholding for pseudo-labeling, the proposed method effectively discovers new intents while maintaining performance on known ones.

Introduction

In practical dialogue systems, users often express intents that fall outside the system's predefined categories. Detecting these "new intents" is essential for system maintenance and business intelligence. However, this task is hindered by the subjectivity of intent definitions and the sparsity of labeled data. Traditional clustering methods often require extensive feature engineering, which is labor-intensive and lacks generalizability.

The proposed training and testing framework for discovering new intents.

The authors address these challenges by proposing a framework that "learns how to cluster" from a small labeled subset and applies this knowledge to partition a large unlabeled pool containing both known and unknown intents.

Methodology

The proposed pipeline consists of three main components:

  1. Intent Representation: Sentences are encoded using a pre-trained BERT model. The resulting embeddings are mean-pooled and transformed via a fully connected layer into a compact representation space.
  2. Pairwise Classification with Similarity Loss: The core innovation lies in reframing clustering as a binary classification problem—determining whether two instances belong to the same intent.
  3. Supervised Learning: The model is trained on labeled pairs to maximize similarity for intra-class pairs and minimize it for inter-class pairs.
  4. Self-Supervised Learning: For unlabeled data, the model uses dynamic thresholds ($u$ for positive pairs, $l$ for negative pairs). These thresholds are adjusted during training, allowing the model to bootstrap its understanding of similarity.
  5. Cluster Refinement: After learning "cluster-friendly" representations, K-means is applied to generate the final intent groupings.

Experimental Setup

The framework was evaluated on three datasets: SNIPS (voice assistant intents), DBPedia (ontology classification), and StackOverflow (technical questions). To simulate a discovery scenario, 25% of the classes were held out as "unknown," and only 10% of the remaining "known" classes were provided with labels during training.

Results and Discussion

The proposed method demonstrates significant improvements over both unsupervised and semi-supervised baselines.

Performance comparison across three datasets showing NMI and Accuracy (ACC) metrics.

As shown in the table above, the model achieves state-of-the-art results:

  • SNIPS: Reached 93.0% Accuracy (ACC), outperforming the best semi-supervised baseline (86.9%).
  • StackOverflow: On this challenging dataset, the method achieved 71.5% ACC, a substantial gain over the unsupervised K-means baseline (13.6%).

The high Normalized Mutual Information (NMI) scores across all datasets indicate that the model successfully organizes unknown intents into coherent clusters rather than simply memorizing known patterns.

Conclusion

This research highlights the power of treating clustering as a pairwise classification task. By leveraging BERT and dynamic thresholding, the model generalizes the concept of "intent similarity" to discover novel categories with minimal supervision. Future work includes exploring end-to-end training architectures and methods for automatically estimating the number of clusters in open-world scenarios.

Summary updated

Abstract

Discovering new user intents is an emerging task in the dialogue system. In this paper, we propose a self-supervised clustering method that can naturally incorporate pairwise constraints as prior knowledge to guide the clustering process and does not require intensive feature engineering. Extensive experiments on three benchmark datasets show that our method can yield significant improvements over strong baselines.
BibTeX
@inproceedings{lin2020constrained,
  author = {Ting-En Lin and Hua Xu and Hanlei Zhang},
  title = {{Constrained Self-Supervised Clustering for Discovering New Intents (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i10.7204},
}
Figure