Dialogue Management in Vector-Based Call Routing

Jennifer Chu-Carroll, Bob CarpenterACL 1998

TL;DR

This paper introduces a novel, domain-independent call routing system that uses vector-based information retrieval to understand spoken customer requests and direct them to the appropriate destination. It achieves a remarkable 93.8% correct routing rate by automatically learning from transcribed calls and dynamically generating disambiguation queries when needed.

What Problem Does This Paper Address?

Traditional call centers often rely on rigid, touch-tone hierarchical menus to route customer calls. This system is inefficient and frustrating for users, requiring them to navigate multiple nested options to find the right department or service. Even simple requests can involve a lengthy menu traversal. While human operators can handle natural language requests, they are expensive and can be a bottleneck. This paper aims to bridge this gap by creating an automatic call router that understands natural spoken language, mimicking the flexibility of a human operator while leveraging the scalability of automated systems.

Methods

The core of the system is a two-component architecture: a Routing Module and a Disambiguation Module.

  • Routing Module: This module treats call routing as a document routing problem. It constructs "virtual documents" for each destination from transcribed calls and represents both these documents and incoming caller requests as vectors in a high-dimensional space.
  • Training: The system first processes a corpus of hand-routed calls. It extracts root forms of words, filters out noise and non-discriminatory terms (using custom ignore and stop lists), and builds n-gram terms. A term-document frequency matrix is created, which is then weighted using inverse document frequency (IDF) to emphasize more specific terms.
  • Vector Representation: Singular Value Decomposition (SVD) is applied to reduce the dimensionality of the term-document matrix, creating vector representations for both terms and destinations.
  • Routing: An incoming caller request is converted into a "pseudo-document vector." The cosine similarity between this vector and the destination vectors is calculated. This similarity score is then transformed using a sigmoid function, fitted to training data, to produce a confidence score for each potential destination. A threshold is used to determine candidate destinations.
  • Disambiguation Module: If the routing module identifies multiple potential destinations (or none), this module is invoked.
  • Query Generation: It dynamically generates disambiguation queries. It selects terms that are "close" to the difference between the request vector and candidate destination vectors, then filters these for "relevance" (terms that can form valid n-grams with existing request terms) and "disambiguating power." Based on the selected terms, it generates either a "wh-question" (e.g., "for what type of loan?") or a "yes-no question" (e.g., "is this about an existing car loan?").
  • Response Handling: The system then processes the caller's response to the query, augmenting the original request and re-routing.

Datasets

  • Training Data: A corpus of 3753 transcribed telephone calls, hand-routed to one of 23 destinations within a financial services call center.
  • Evaluation Data:
    • Routing Module: 389 transcribed calls, disjoint from the training set. This set included 307 unambiguous calls and 82 ambiguous calls.
  • Disambiguation Module: 157 calls that were initially ambiguous and required a follow-up question from a human operator. This set was also disjoint from the training and routing evaluation sets.

Key Insights and Ideas

  • Vector Space Models for Dialogue: Applying vector space models (like those used in document retrieval) to spoken dialogue for routing is a powerful and effective approach.
  • Dynamic Query Generation: The ability to dynamically generate context-specific disambiguation queries, rather than relying on pre-scripted menus, significantly improves the system's flexibility and naturalness.
  • Domain Independence: The core vector-based approach and the disambiguation mechanism are designed to be domain-independent, meaning the system can be ported to new domains with minimal modification, requiring only new training data.
  • Sigmoid Confidence Fitting: Directly mapping raw cosine similarity scores to confidence scores using a sigmoid function, tailored per destination, is crucial for achieving optimal routing performance. Raw similarity doesn't always translate directly to routing likelihood.

Key Results

  • Overall Performance: The system achieved an overall correct routing rate of 93.8% for automatically routed calls, while rejecting (sending to a human operator) 10.2% of calls.
  • Routing Module Performance: On unambiguous calls, the routing module achieved 93.5% correct routing using sigmoid confidence fitting, a 16.7% relative reduction in error rate compared to using raw cosine scores.
  • Disambiguation Module Performance: For ambiguous calls, the system successfully routed 87.0% of those that underwent disambiguation to the correct destination.

Limitations and Future Work

  • Response Understanding: The current system's understanding of yes/no answers is basic (explicitly looking for "yes"). The authors suggest incorporating more sophisticated natural language understanding models for response interpretation.
  • Disjunctive Questions: The system currently generates wh-questions or yes-no questions. Exploring disjunctive questions (e.g., "Is it a new car loan or an existing car loan?") is suggested as a future avenue.
  • Test Set Limitations: For the routing module evaluation, only the first utterance of ambiguous calls was transcribed, limiting the analysis of how ambiguous calls were eventually handled.
  • Speech Recognition Integration: While the paper focuses on transcribed calls, a companion paper (mentioned in the references) compares performance with speech recognizer output, indicating this is an area of ongoing work.

Summary updated

BibTeX
@inproceedings{chucarroll1998dialogue,
  author = {Jennifer Chu-Carroll and Bob Carpenter},
  title = {{Dialogue Management in Vector-Based Call Routing}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {1998},
}
Figure