TL;DR
This paper introduces a framework (ENCS) to quantify the economic trade-offs of using Large Language Models (LLMs) for customer service agent assistance. They found that while model quality is crucial, the inference cost of LLMs can significantly impact overall savings, suggesting that smaller, distilled models might offer better ROI than larger, more powerful ones in certain scenarios.
What Problem Does This Paper Address?
Customer service is expensive, and companies are increasingly looking to AI to assist human agents. Large Language Models (LLMs) show promise in generating helpful responses, but their high training and serving costs present a significant economic challenge. The paper addresses the question: how can businesses effectively evaluate the economic viability of integrating LLMs into agent assistance tools, considering both the quality of LLM-generated responses and their associated costs? They aim to provide a framework for making informed decisions about which LLM strategies (e.g., prompt engineering, fine-tuning, distillation) offer the best return on investment.
Methods
The authors propose and utilize an Expected Net Cost Savings (ENCS) framework. This framework quantifies savings by considering:
- The probability of an agent using, editing, or ignoring an LLM-generated response.
- The estimated time savings (or cost) associated with each action.
- The cost of generating the LLM response.
They explore three LLM adaptation strategies:
- Prompt Engineering: Crafting effective prompts to guide LLMs without modifying their weights.
- Fine-Tuning: Adapting pre-trained LLMs to specific conversational data.
- Knowledge Distillation: Training a smaller "student" model to mimic the behavior of a larger "teacher" LLM, aiming to reduce inference costs.
They also use perplexity as a proxy for response quality and investigate its correlation with agent usability.
Datasets
- Brand-Specific Training and Test Sets: Collected from a single retail brand (Anonymous Retailer - AR), comprising customer service conversations from 2022. The test set was manually curated for specific scenarios.
- General Training Set: Constructed from five additional retail brands with similar product lines, used for more generalizable fine-tuning and distillation.
Key Insights and Ideas
- "Bigger isn't always better": A distilled GPT-2 model, despite potentially lower response quality than GPT-3, yielded greater cost savings due to its significantly lower inference cost.
- Usability Trumps Raw Quality (to an extent): The usability of a model's responses (how likely an agent is to use or edit it) can compensate for large differences in inference cost.
- Perplexity as a Predictor: Lower perplexity generally correlates with higher agent response usage, providing a way to estimate usability and cost savings for models not manually evaluated.
- Inference Cost is a Major Factor: For high-traffic brands, the cost of serving LLM responses can outweigh the benefits of slightly higher quality, making distilled models more attractive.
- R&D vs. Inference Cost: While inference cost is critical for ongoing savings, R&D cost is a primary driver for reaching the break-even point, especially for smaller brands.
Key Results
- Agent Usage: Agents used suggested responses 57-69% of the time, with GPT-3 PE showing the highest usage (69%).
- Cost Savings: The distilled GPT-2 model (GPT-2 BFT BD) achieved an estimated annual cost saving of $53,653 for the case-study brand, while GPT-3 PE achieved $50,920. GPT-3 BFT resulted in a loss of $18,691 annually due to high inference costs.
- Perplexity Correlation: A statistically significant relationship was found between perplexity and agent response usability (use, edit, ignore) and foundation metrics (sensible, specific, informative, helpful, role-consistent).
- Break-Even Point: For a high-traffic brand, models could break even in as little as 0.24 months (in-house) or 0.29 months (3rd party), highlighting the speed at which savings can accrue.
Limitations and Future Work
- Limited Annotators: The study relied on a small number of agents (9) for manual annotation, and the linear models were based on a relatively small test set (287 conversations).
- Agent Preferences: The study acknowledges that agent preferences can influence usability scores, and real-world metrics (like AHT) might incentivize different behavior.
- Workforce Optimization: The ENCS framework simplifies by assuming agents are always busy, not accounting for potential workforce reductions.
- Risk of Errors: The paper omits the cost of agents using inappropriate or factually incorrect responses, which is noted as a critical area for future study.
- Generalizability: While the framework is designed to be general, the specific findings are from a single brand case study.
- Future Work: The authors suggest further analysis of risk and cost of errors, and expanding the study to a broader range of brands and models. They also emphasize the need for continuous adaptation of the ENCS framework as LLM technology and pricing evolve.
Summary updated