A data visualization and analysis method for natural language call routing system design

Hong-Kwang Jeff Kuo, Vaibhava GoelInterspeech 2007

TL;DR

This paper introduces a novel data visualization and analysis tool designed to help engineers build better natural language call routing systems. By automatically clustering caller utterances and highlighting potential confusions between routing targets, the tool significantly streamlines the process of identifying and correcting labeling errors and optimizing dialogue design.

What Problem Does This Paper Address?

Designing effective speech-enabled call routing systems is crucial for large customer contact centers, but it's a complex process. Engineers often face challenges in understanding the vast amounts of caller data, ensuring consistent labeling, and adapting to changes in business logic. Existing methods focus on implementing routing systems or improving classification, but there's a gap in tools that help visualize and analyze the labeled data itself to inform design decisions. This paper addresses the need for a system that can help designers gain insights from this data, identify inconsistencies, and proactively design for inherent ambiguities in caller requests.

Methods

The core of the proposed solution is a data visualization and analysis tool that leverages automatic clustering and similarity analysis.

  • Feature Vector Generation: The process begins by training a maximum entropy (maxent) classification model on the labeled training data. Crucially, the parameters (weights) learned by this model are used to create feature vectors for each sentence. These weights effectively highlight semantically important words for routing, de-emphasizing less relevant ones, without needing explicit stop word lists.
  • Clustering: For each target class (e.g., "Technical Support," "Billing Inquiry"), sentences are clustered into homogeneous groups using a K-means algorithm. The similarity metric used is the dot product of the weighted feature vectors. A heuristic initialization procedure is employed to find initial cluster centroids.
  • Confusion Detection: After clustering within each target class, the tool computes pair-wise similarity between the centroids of clusters from different target classes. High similarity between clusters from distinct targets indicates potential confusion, suggesting either labeling errors or inherent ambiguity in caller requests.
  • Visualization: The tool provides a multi-level interface:
  • A Main View shows targets, counts of unique and total sentences, and the number of clusters found per target.
  • A Cluster Level View displays prototype sentences for each cluster, along with the top features and their weights. It also highlights confusable clusters from other targets.
  • A Sentence Level View allows drilling down to see individual sentences within a cluster.

Datasets

The authors applied their tool to data from call centers of major Fortune 500 companies. The specific example presented uses data collected from a Wizard-of-Oz system for a major financial services company, comprising approximately 44,000 sentences manually labeled into 40 routing targets. No new benchmarks were introduced; the focus is on analyzing existing labeled data.

Key Insights and Ideas

  • Leveraging Classifier Weights for Clustering: Using parameters from a trained classifier to weight word importance in feature vectors is a clever way to implicitly handle stop words and focus on semantically relevant terms for clustering, without manual intervention.
  • Clustering as a Data Relabeling Aid: The ability to relabel entire clusters instead of individual sentences dramatically improves the efficiency of correcting erroneous labels or adapting to business logic changes.
  • Proactive Ambiguity Detection: Identifying inherently confusable clusters before deployment allows designers to proactively build disambiguation dialogues, leading to a more robust and user-friendly system.
  • Bridging Top-Down and Bottom-Up Design: The visualization tool effectively bridges the gap between high-level business constraints (top-down) and the actual caller utterances (bottom-up), enabling more informed design decisions.

Key Results

The paper demonstrates the utility of the tool through an example from a financial services company. The visualization allowed for:

  • Efficient Data Exploration: Users can navigate from a high-level overview of targets to individual sentences with a few clicks.
  • Identification of Labeling Inconsistencies: The tool helped uncover specific instances of mislabeled sentences (e.g., "transfer to a 401k account" mislabeled as TRANSFER CALL instead of EXCHANGE).
  • Highlighting Ambiguous Requests: The tool identified confusable clusters for requests like "i need some money," which could relate to loans or withdrawals, indicating the need for disambiguation dialogues.
  • Streamlined Business Logic Updates: The ability to manipulate clusters (move, create, delete) makes it easier to update the routing system when business logic changes.

Limitations and Future Work

  • Semantic Similarity: The current clustering procedure does not account for semantic similarity between words (e.g., "fund" vs. "funds," "operator" vs. "representative"). Future work could incorporate techniques to address this.
  • Variations in Clustering: The authors suggest further experiments with variations in the clustering procedure, similarity metrics, and prototype vector definitions to improve results.
  • Broader Applicability: While demonstrated for call routing, the authors believe similar visualization tools could benefit other natural language applications.

Summary updated

BibTeX
@inproceedings{kuo2007data,
  author = {Hong-Kwang Jeff Kuo and Vaibhava Goel},
  title = {{A data visualization and analysis method for natural language call routing system design}},
  booktitle = {Interspeech},
  year = {2007},
}
Figure