Probing the Depths of Language Models’ Contact-Center Knowledge for Quality Assurance

Digvijay Anil Ingle, Aashraya Sachdeva, Surya Prakash Sahu, Mayank Sati, Cijo George, Jithendra VepaEMNLP 2024

TL;DR

This paper investigates how well large language models (LLMs) can perform Quality Assurance (QA) for contact-center conversations. They found that larger LLMs perform better and propose methods to transfer this domain knowledge to smaller models, achieving significant performance improvements. This work is crucial for developing more objective and scalable AI-assisted tools for evaluating customer service agents.

What Problem Does This Paper Address?

Contact centers are increasingly leveraging AI to improve customer service quality and operational efficiency. A key aspect of this is evaluating agent performance through Quality Assurance (QA). This QA process traditionally involves human analysts meticulously reviewing conversations to ensure agents adhere to policies, exhibit good behavior, and provide effective service. However, this manual process is time-consuming, cognitively demanding, and prone to inconsistencies. While LLMs have shown remarkable capabilities in natural language understanding, their application to the nuanced domain of contact-center QA, which requires deep domain knowledge and linguistic understanding, remains under-explored. This paper aims to bridge this gap by evaluating LLMs' out-of-the-box performance on contact-center QA and proposing methods to distill domain knowledge into smaller, more efficient models.

Methods

The authors propose a multi-faceted approach:

  • Comprehensive LLM Evaluation: They benchmark eight different LLMs (categorized as Large, Medium, and Small based on parameter count) on a custom contact-center QA dataset. The evaluation focuses on the LLMs' ability to answer QA questions based on conversation transcripts, requiring them to identify evidence, synthesize it, and provide a reasoned answer.
  • Domain Knowledge Distillation: To address the performance gap between large and small LLMs, they explore two primary methods:
  • Evaluation Plan Generation: A larger, more knowledgeable LLM (Claude-3.5-Sonnet) is used to generate a structured "evaluation plan" for a given QA question. This plan breaks down the question into simpler, actionable criteria. This plan is then used to guide the reasoning of a smaller LLM.
  • Fine-tuning with Generated Data: The smaller LLM is fine-tuned using "silver data" (responses generated by a larger LLM) that includes both the evaluation plan and the reasoned answer. This approach aims to transfer the domain-specific knowledge without requiring extensive human annotation.
  • Chain-of-Thought (CoT) Reasoning: Throughout their experiments, they leverage CoT prompting to encourage LLMs to provide step-by-step reasoning, mirroring human annotation processes and enhancing transparency.

Datasets

  • Custom QA Dataset (DQA): The authors curated a specialized dataset for contact-center QA. This dataset consists of 100 English dyadic conversations and 50 proprietary QA questions. Seven annotators, experts in contact-center QA, were used to label question-conversation pairs, resulting in a refined dataset of 3,061 pairs with ground-truth answers and reasoning.
  • Training Dataset (DTrain): A separate dataset was created for fine-tuning, with a balanced distribution of "yes" and "no" labels, ensuring the model doesn't develop a bias towards one answer.

Key Insights and Ideas

  • Model Size Matters: Larger LLMs consistently outperform smaller ones in contact-center QA, highlighting the importance of their extensive pre-training and inherent domain knowledge.
  • Evaluation Plans are Crucial: Generating an explicit evaluation plan, even from a larger LLM, significantly boosts the performance of smaller LLMs. This plan acts as a structured guide, helping smaller models focus on relevant aspects of the conversation.
  • Distillation is Effective: Transferring domain knowledge from large to small LLMs through generated evaluation plans and fine-tuning with silver data is a viable strategy. This approach can significantly improve smaller models' performance without the need for costly human annotation.
  • Human-in-the-Loop Potential: The evaluation plan framework naturally lends itself to human-in-the-loop refinement, suggesting a path towards even more sophisticated and accurate AI-assisted QA systems.

Key Results

  • Out-of-the-box Performance: Claude-3.5-Sonnet achieved the highest Macro F1 score of 75.48% among the evaluated LLMs.
  • Knowledge Distillation Improvement:
    • Using an evaluation plan from a larger LLM improved the performance of Phi-3-mini-128k-instruct by over 5% compared to its out-of-the-box performance.
  • Fine-tuning Phi-3-mini-128k-instruct with responses generated by Claude-3.5-Sonnet resulted in a substantial improvement of 18.95% over its baseline performance.
  • Ablation Study: The results indicated that the evaluation plan plays a critical role, with a marginal drop in performance when fine-tuning without it.

Limitations and Future Work

  • ASR Dependency: The current system relies on ASR transcripts, and the performance is sensitive to the Word Error Rate (WER) of the ASR engine. The study specifically used US-English dialects, and its applicability to other dialects is not discussed.
  • Ethical Considerations: The paper acknowledges potential biases in ASR systems and emphasizes the need for human oversight, transparency, and data security.
  • Future Work: The authors suggest exploring the interpretability of generated reasoning, further investigating inference parameter sensitivity for models like GPT-40-mini, and developing more robust human-in-the-loop feedback mechanisms to refine evaluation plans and potentially surpass larger models' capabilities. They also highlight the need for further research into the critical role of evaluation plans in guiding model reasoning.

Summary updated

BibTeX
@inproceedings{ingle2024probing,
  author = {Digvijay Anil Ingle and Aashraya Sachdeva and Surya Prakash Sahu and Mayank Sati and Cijo George and Jithendra Vepa},
  title = {{Probing the Depths of Language Models’ Contact-Center Knowledge for Quality Assurance}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
}
Figure