AI Coach Assist: An Automated Approach for Call Recommendation in Contact Centers for Agent Coaching

Md Tahmid Rahman Laskar, Cheng Chen, Xue-yong Fu, Mahsa Azizi, Shashi Bhushan, Simon Corston-oliverACL 2023

TL;DR

This paper introduces "AI Coach Assist," a novel system that uses transformer-based language models to automatically identify customer service calls most relevant for agent coaching. By analyzing call transcripts and specific quality assurance questions, the system helps supervisors prioritize their coaching efforts, moving beyond random call selection to more targeted and effective agent development.

What Problem Does This Paper Address?

Contact centers face a deluge of customer calls daily, making it incredibly challenging for managers and supervisors to manually sift through recordings to identify calls that offer the best coaching opportunities. Traditional methods often rely on random sampling, which is inefficient and can lead to missed coaching moments or wasted resources on calls that don't require improvement. This gap hinders effective agent development and, consequently, customer satisfaction. The paper aims to bridge this by automating the identification of "coachable" calls.

Methods

The core of AI Coach Assist is a text classification model that takes a quality assurance (QA) question posed by a manager and a corresponding call transcript as input. The system leverages pre-trained transformer-based language models, specifically fine-tuning them for this pairwise sentence classification task.

The authors explore two main model choices:

  • DistilBERT: Chosen for its efficiency, being a smaller and faster distilled version of BERT, making it suitable for production environments with limited computational resources.
  • DialogLED: Selected for its potential to achieve higher accuracy, as it's pre-trained on long dialogue conversations and can handle longer input sequences, which is crucial given the length of typical contact center transcripts.

The input to the model is created by concatenating the QA question and the call transcript, allowing the transformer to learn the contextual relationship between them. The model then outputs a probability score indicating the likelihood of the call being coachable for that specific question.

Datasets

The paper introduces a new benchmark dataset called QA Scorecard. This dataset was collected from real-world contact centers and consists of customer-agent call transcripts generated by Automatic Speech Recognition (ASR) systems. Crucially, these transcripts are annotated by actual contact center managers and supervisors who assessed the calls based on specific QA questions. A call is deemed "coachable" for a particular question if it scores less than 50% on that question. The dataset was curated over a year and includes diverse call types. The authors also ensured a balanced class distribution (coachable vs. not coachable) for each question to avoid bias.

Key Insights and Ideas

  • Contextualized Coaching: The fundamental idea is that a specific QA question provides crucial context for determining if a call is coachable. Simply analyzing a transcript in isolation is less effective than doing so with a targeted question in mind.
  • Transformer Power for Dialogue: Fine-tuning transformer models, especially those pre-trained on dialogue data like DialogLED, is highly effective for understanding the nuances of customer-agent interactions in contact centers.
  • Length Matters: The ability of models like DialogLED to handle longer sequences is a significant advantage, as many contact center transcripts exceed the typical 512-token limit of models like BERT and DistilBERT. This longer context is vital for accurate assessment.
  • Efficiency vs. Accuracy Trade-off: The study highlights a clear trade-off between model efficiency (DistilBERT) and accuracy (DialogLED), allowing for deployment choices based on specific operational needs.

Key Results

  • DialogLED Outperforms: The DialogLED model consistently outperformed DistilBERT across all metrics (precision, recall, F1, and accuracy) on the QA Scorecard dataset, achieving an accuracy of 70.52%.
  • Query is Crucial: Ablation studies showed a significant drop in accuracy (up to 10.90% for DialogLED) when the QA question was removed from the input, underscoring its importance.
  • Longer Sequences Benefit: Reducing the input sequence length for DialogLED led to substantial drops in accuracy and precision, demonstrating the value of its long-context processing capability.
  • Superior to Baselines: Both DialogLED and DistilBERT significantly outperformed traditional TF-IDF based machine learning models, with DialogLED showing a 17.78% accuracy improvement on average.
  • Question Type Variability: Performance varied across different question types, with DialogLED excelling in Greeting and Account Verification but facing challenges with Behavioral and Issue Resolution questions.

Limitations and Future Work

  • Language Dependency: The current models are trained on English conversations and may not be directly applicable to other languages or input types.
  • Question Type Specificity: The system's performance varies by question type, meaning users need to carefully select which question types to deploy.
  • Computational Resources: DialogLED, while more accurate, requires higher computational resources than DistilBERT.
  • Future Directions: The authors plan to improve performance on challenging question types, explore other question-answering models, and investigate generative LLMs to provide reasons for coachability. They also aim to prototype the system for real-world use, focusing on ethical considerations and preventing misuse, such as automating performance reviews or using biased questions.

Summary updated

BibTeX
@inproceedings{laskar2023ai,
  author = {Md Tahmid Rahman Laskar and Cheng Chen and Xue-yong Fu and Mahsa Azizi and Shashi Bhushan and Simon Corston-oliver},
  title = {{AI Coach Assist: An Automated Approach for Call Recommendation in Contact Centers for Agent Coaching}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure