Improving Hierarchical Text Clustering with LLM-guided Multi-view Cluster Representation

Anup Pattnaik, Cijo George, Rishabh Kumar Tripathi, Sasanka Vutla, Jithendra VepaEMNLP 2024

TL;DR

This paper introduces a novel hierarchical text clustering approach for contact center interaction drivers, leveraging Large Language Models (LLMs) to create multi-view representations of lower-level clusters. This LLM-guided approach significantly improves the quality of higher-level clusters, aligning them better with human preferences and achieving state-of-the-art results on both internal and public datasets.

What Problem Does This Paper Address?

Contact centers generate vast amounts of text data from customer interactions. Identifying "interaction drivers" – the core reasons for these interactions – is crucial for business analytics, quality assurance, and gaining insights. While automated methods exist, accurately clustering these drivers, especially into a hierarchical structure (e.g., broad categories and sub-categories), remains challenging. Current methods often fail to capture the nuanced semantic abstractions that humans naturally use, leading to higher-level clusters that don't align well with business needs. The paper aims to bridge this gap by using LLMs to enhance the representation of lower-level clusters, thereby improving the quality and human-alignment of the higher-level hierarchical clusters.

Methods

The proposed approach is a multi-stage hierarchical clustering method:

  1. Deriving L2 Clusters: Standard agglomerative clustering is first applied to the raw interaction documents to generate initial, lower-level clusters (L2).
  2. Sampling Representative Documents: For each L2 cluster, a subset of representative documents is selected using Maximal Marginal Relevance (MMR). This balances relevance to the cluster's theme with diversity among the selected documents.
  3. LLM-guided Multi-view Representations: This is the core innovation. For each L2 cluster, three "views" are generated:
  4. Centroid Embedding: The average embedding of all documents in the L2 cluster.
  5. Name Embedding: An embedding of a concise, LLM-generated cluster name (3-5 words).
  6. Description Embedding: An embedding of a more detailed, LLM-generated cluster description (under 50 words). These three views are then combined into a single multi-view representation using a weighted average. The LLM used is a fine-tuned Llama-3 Instruct 8B model.

  7. Generating L1 Clusters: The weighted multi-view representations of the L2 clusters are then fed into another agglomerative clustering step to form the higher-level clusters (L1). LLM-generated names and descriptions are also created for these L1 clusters for explainability.

The approach is also adapted for standard non-hierarchical clustering by treating labeled classes as L1 clusters and assuming a hidden layer of L2 clusters.

Datasets

  • Internal Datasets: Three custom datasets were created from 5,000 real contact center interactions each, covering Quick Commerce, Education, and Travel domains. These datasets were generated by an in-house LLM and sampled from live chat and call transcripts. They are not publicly released due to proprietary reasons.
  • Public Datasets: The approach was also evaluated on two standard labeled intent classification datasets:
  • Banking77: 13,083 customer service queries with 77 intents.
  • CLINC150: 23,700 samples across 10 domains with 150 intents. For these, the labeled intents were treated as L1 clusters, and a hidden layer of 500 L2 clusters was assumed.

The authors also open-source two new labeled datasets for two-level hierarchical clustering derived from Banking77 and CLINC150.

Key Insights and Ideas

  • LLMs as Semantic Enhancers: The paper demonstrates that LLMs can effectively capture abstract semantic information, which is crucial for improving the quality of higher-level clusters in hierarchical structures.
  • Multi-view Representation is Key: Combining different "views" (centroid, name, description) of L2 clusters significantly outperforms using any single view, highlighting the benefit of a holistic representation.
  • Human Alignment is Achievable: The LLM-guided approach leads to L1 clusters that are demonstrably better aligned with human preferences, a critical factor for practical business applications.
  • Cost-Effectiveness: The method requires a limited number of LLM queries (proportional to the number of L2 clusters, not the total documents), making it more cost-effective than other LLM-guided approaches.

Key Results

  • Internal Datasets:
    • Average Silhouette Scores improved by up to 70%.
  • Human Preference Scores increased by an average of 36.7%.
  • Public Datasets (Non-Hierarchical Adaptation):
    • Improved NMI scores by 10.3% and 9.2% over standard agglomerative clustering.
  • Improved ACC scores by 11.3% and 11.7%.
  • Achieved state-of-the-art NMI (94.2) and ACC (86.2) scores on CLINC150.
  • Performance was robust across varying numbers of L2 clusters.

Limitations and Future Work

  • Clustering Algorithm Independence: While the current work uses agglomerative clustering, the methodology is adaptable to other algorithms like k-means, DBSCAN, or spectral clustering.
  • LLM Choice: The paper used a specific external LLM. Future work could explore different LLM architectures and sizes to find optimal configurations for generating cluster names and descriptions.
  • Optimal L2 Cluster Number: Determining the optimal number of L2 clusters in unsupervised settings remains a challenge. Future research could focus on developing more efficient methods for this.
  • Dataset Release: The authors open-source two new hierarchical clustering datasets derived from Banking77 and CLINC150 to benefit the research community.

Summary updated

BibTeX
@inproceedings{pattnaik2024improving,
  author = {Anup Pattnaik and Cijo George and Rishabh Kumar Tripathi and Sasanka Vutla and Jithendra Vepa},
  title = {{Improving Hierarchical Text Clustering with LLM-guided Multi-view Cluster Representation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
}
Figure