TL;DR
This paper introduces a novel hierarchical text clustering approach for contact center interaction drivers, leveraging Large Language Models (LLMs) to create multi-view representations of lower-level clusters. This LLM-guided approach significantly improves the quality of higher-level clusters, aligning them better with human preferences and achieving state-of-the-art results on both internal and public datasets.
What Problem Does This Paper Address?
Contact centers generate vast amounts of text data from customer interactions. Identifying "interaction drivers" – the core reasons for these interactions – is crucial for business analytics, quality assurance, and gaining insights. While automated methods exist, accurately clustering these drivers, especially into a hierarchical structure (e.g., broad categories and sub-categories), remains challenging. Current methods often fail to capture the nuanced semantic abstractions that humans naturally use, leading to higher-level clusters that don't align well with business needs. The paper aims to bridge this gap by using LLMs to enhance the representation of lower-level clusters, thereby improving the quality and human-alignment of the higher-level hierarchical clusters.
Methods
The proposed approach is a multi-stage hierarchical clustering method:
- Deriving L2 Clusters: Standard agglomerative clustering is first applied to the raw interaction documents to generate initial, lower-level clusters (L2).
- Sampling Representative Documents: For each L2 cluster, a subset of representative documents is selected using Maximal Marginal Relevance (MMR). This balances relevance to the cluster's theme with diversity among the selected documents.
- LLM-guided Multi-view Representations: This is the core innovation. For each L2 cluster, three "views" are generated:
- Centroid Embedding: The average embedding of all documents in the L2 cluster.
- Name Embedding: An embedding of a concise, LLM-generated cluster name (3-5 words).
-
Description Embedding: An embedding of a more detailed, LLM-generated cluster description (under 50 words). These three views are then combined into a single multi-view representation using a weighted average. The LLM used is a fine-tuned Llama-3 Instruct 8B model.
-
Generating L1 Clusters: The weighted multi-view representations of the L2 clusters are then fed into another agglomerative clustering step to form the higher-level clusters (L1). LLM-generated names and descriptions are also created for these L1 clusters for explainability.
The approach is also adapted for standard non-hierarchical clustering by treating labeled classes as L1 clusters and assuming a hidden layer of L2 clusters.
Datasets
- Internal Datasets: Three custom datasets were created from 5,000 real contact center interactions each, covering Quick Commerce, Education, and Travel domains. These datasets were generated by an in-house LLM and sampled from live chat and call transcripts. They are not publicly released due to proprietary reasons.
- Public Datasets: The approach was also evaluated on two standard labeled intent classification datasets:
- Banking77: 13,083 customer service queries with 77 intents.
- CLINC150: 23,700 samples across 10 domains with 150 intents. For these, the labeled intents were treated as L1 clusters, and a hidden layer of 500 L2 clusters was assumed.
The authors also open-source two new labeled datasets for two-level hierarchical clustering derived from Banking77 and CLINC150.
Key Insights and Ideas
- LLMs as Semantic Enhancers: The paper demonstrates that LLMs can effectively capture abstract semantic information, which is crucial for improving the quality of higher-level clusters in hierarchical structures.
- Multi-view Representation is Key: Combining different "views" (centroid, name, description) of L2 clusters significantly outperforms using any single view, highlighting the benefit of a holistic representation.
- Human Alignment is Achievable: The LLM-guided approach leads to L1 clusters that are demonstrably better aligned with human preferences, a critical factor for practical business applications.
- Cost-Effectiveness: The method requires a limited number of LLM queries (proportional to the number of L2 clusters, not the total documents), making it more cost-effective than other LLM-guided approaches.
Key Results
- Internal Datasets:
- Average Silhouette Scores improved by up to 70%.
- Human Preference Scores increased by an average of 36.7%.
- Public Datasets (Non-Hierarchical Adaptation):
- Improved NMI scores by 10.3% and 9.2% over standard agglomerative clustering.
- Improved ACC scores by 11.3% and 11.7%.
- Achieved state-of-the-art NMI (94.2) and ACC (86.2) scores on CLINC150.
- Performance was robust across varying numbers of L2 clusters.
Limitations and Future Work
- Clustering Algorithm Independence: While the current work uses agglomerative clustering, the methodology is adaptable to other algorithms like k-means, DBSCAN, or spectral clustering.
- LLM Choice: The paper used a specific external LLM. Future work could explore different LLM architectures and sizes to find optimal configurations for generating cluster names and descriptions.
- Optimal L2 Cluster Number: Determining the optimal number of L2 clusters in unsupervised settings remains a challenge. Future research could focus on developing more efficient methods for this.
- Dataset Release: The authors open-source two new hierarchical clustering datasets derived from Banking77 and CLINC150 to benefit the research community.
Summary updated