“Are you calling for the vaporizer you ordered?” Combining Search and Prediction to Identify Orders in Contact Centers

Abinaya K, Shourya RoyACL 2021

TL;DR

This paper tackles the challenge of quickly identifying which of a customer's many active orders they're calling about in an e-commerce contact center. They propose a two-stage approach combining fuzzy search and predictive modeling, achieving 64% coverage with 87% accuracy on real-world data, significantly improving upon existing methods and demonstrating practical deployment benefits.

What Problem Does This Paper Address?

In the booming world of e-commerce, customer satisfaction hinges on efficient contact center operations. A major bottleneck is identifying the specific order a customer is calling about, especially when they have multiple active orders. Traditional methods, like enumerating orders and asking the customer to select one, are tedious, time-consuming, and limited in scope. This paper addresses the critical need for an automated, accurate, and efficient way for voice bots to pinpoint the customer's order of interest, thereby reducing call duration and enhancing customer experience.

Methods

The authors propose a novel two-stage approach:

  1. Fuzzy Search: This stage acts like a retrieval-based question-answering system. It takes the customer's transcribed utterance as a query and searches through the customer's active orders. The novelty lies in the custom matching techniques that leverage both lexical (word-based) and phonetic (sound-based) similarities. This is crucial for handling the noisy and often non-standard language found in ASR (Automatic Speech Recognition) transcripts. The process involves several steps:
  2. Pre-processing: Removing generic filler words and handling split words (e.g., "t v" for "TV").
  3. Direct Match: Exact text matching between query and product title tokens.
  4. Partial Match: Using n-grams (sequences of words) to find matches even with partial product names or ASR errors.
  5. Phonetic Match: Employing phonetic representations to match utterances that sound similar but are spelled differently due to ASR noise (e.g., "mam record" vs. "memory card").
  6. Order Prediction: This stage complements fuzzy search by predicting the most likely order a customer is calling about, even without explicit utterance clues. It uses a supervised learning model (specifically, a Random Forest model is highlighted) trained on various features. These features are categorized as:
  7. Order-specific: Status, delivery due, refund issued.
  8. Transaction-specific: Price, shipping charges, payment type.
  9. Product-specific: Brand, vertical, item size.
  10. Self-serve related: Days since last chat, days since last incident. Crucially, the model also incorporates derived features that capture the relative ordering and relationships between a customer's active orders (e.g., rank by order date, presence of existing complaints for other orders).

Datasets

  • Search Dataset: A development set of ~2.5K calls and a large test set of ~95K calls were used for training and evaluating the fuzzy search technique. This dataset consists of transcribed customer utterances and the product attributes of their active orders.
  • Prediction Dataset: A dataset of 150K customer calls was used for training and evaluating the order prediction models. This dataset includes features of active orders and the customer's chosen order as ground truth.

Key Insights and Ideas

  • The Power of Phonetics for ASR Noise: The paper highlights that ASR transcripts are inherently noisy. Phonetic matching is a key innovation that allows the system to bridge the gap between how a customer says a product name and how it's transcribed, significantly improving matching accuracy.
  • Beyond Keywords: Predictive Modeling Matters: Relying solely on search can miss a significant portion of calls where customers don't provide clear cues. The predictive model, by leveraging order and customer history features, effectively captures implicit signals, boosting overall coverage.
  • Context is King: Derived Features: The insight that a customer's intent is influenced by their entire order history is crucial. Incorporating derived features that capture these inter-order relationships is a sophisticated way to improve prediction accuracy.
  • Customization Beats Generalization: The authors argue that their custom fuzzy search, tailored for the specific domain of e-commerce orders and ASR noise, outperforms general-purpose e-commerce search (which relies on NER) and even text embedding-based similarity models.

Key Results

  • Combined Approach Performance: The proposed two-staged approach (Fuzzy Search + Order Prediction) achieved 64% coverage with 87% accuracy on a large real-life dataset.
  • Fuzzy Search Performance: Achieved 32% coverage with 87% accuracy on the development set.
  • Order Prediction Performance: Achieved 72.52% top-1 accuracy with Random Forest, and significantly improved coverage when combined with search.
  • Incremental Improvement: Order prediction added an incremental coverage of 44% while maintaining the accuracy of the search-only approach.
  • Practical Deployment: The system has been deployed in production for a fraction of calls, demonstrating real-world operational benefits.

Limitations and Future Work

  • Limitations:
    • The fuzzy search coverage is limited by the customer's response; approximately 25% of callers did not answer when asked to identify their order.
  • While the combined approach is strong, there's still a portion of calls that remain unmatched.
  • Future Work:
    • The authors suggest that the order prediction model could be extended to other channels like chatbots or proactive outreach.
  • The paper implies that further refinement of features and models could potentially increase coverage and accuracy.

Summary updated

BibTeX
@inproceedings{k2021are,
  author = {Abinaya K and Shourya Roy},
  title = {{“Are you calling for the vaporizer you ordered?” Combining Search and Prediction to Identify Orders in Contact Centers}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure