Introduction
In virtual assistant development, identifying user "intents" is critical. For instance, "How do I set up direct deposit?" and "I want to switch to direct deposit" both map to a DIRECT_DEPOSIT intent. Traditionally, developers chose between unsupervised clustering (flexible but inaccurate) and supervised classification (accurate but rigid).
This paper explores supervised clustering, where a model learns the underlying logic of grouping from labeled data and applies it to discover entirely new, unseen categories. While previous attempts used linear models like SVMs, they lacked the semantic depth of modern neural networks. Conversely, standard neural networks often focus on local pairwise decisions rather than global cluster structures.
Neural Supervised Clustering (NSC)
The authors propose Neural Supervised Clustering (NSC), a framework that integrates a Latent Structured Prediction loss into a neural pipeline.
Architecture and Inference
NSC utilizes a pairwise encoder (CNN or BERT) to generate similarity scores for all pairs in a dataset. To perform inference, these scores are treated as edge weights in a graph where questions are nodes. The model employs Kruskal’s algorithm to find a Maximum Spanning Forest (MSF), where each connected tree represents a distinct cluster.
Global Optimization
Unlike standard binary cross-entropy, NSC uses a max-margin objective derived from Latent Structural SVMs (LSSVM). The model is penalized if the score of an incorrect clustering is too close to the "gold" clustering. This structural loss is sub-differentiable, allowing for end-to-end backpropagation from the global cluster level down to word embeddings.
Experimental Setup
The approach was evaluated on two benchmarks:
- IC&OOS (Intent Classification & Out-of-Scope): Repurposed for clustering by testing on categories never seen during training.
- Quora Intent Corpus: Based on duplicate detection, using both automatic and expert-annotated labels.
Key Results
The NSC framework demonstrates a significant performance leap over traditional methods.
On the IC&OOS dataset, NSC-BERT achieved an F1 score of 95.65%, representing a 17.24% absolute improvement over the best unsupervised baseline (Spectral Clustering). Even with a simpler CNN encoder, the supervised structural approach (89.19% F1) significantly outperformed unsupervised methods (78.41% F1).
Out-of-Scope Detection
A major advantage of NSC is its ability to handle "unknown" queries. By learning clustering logic rather than fixed labels, the model naturally identifies "Out-of-Scope" (OOS) queries as singleton clusters.
NSC-BERT achieved a 71.45% OOS Recall, nearly doubling the effectiveness of standard intent classifiers in identifying queries that do not fit known categories.
Discussion and Future Work
While NSC provides a powerful tool for intent discovery, challenges remain. The $O(n^2)$ complexity of pairwise scoring limits scalability to very large datasets. Additionally, error analysis indicates that the model can still be misled by high word overlap in semantically distinct questions. Future research may focus on scaling these methods and jointly predicting intents alongside specific slot attributes.
Summary updated