TL;DR
This paper introduces "minimally invasive surgery" techniques for optimizing commercial spoken dialog systems. By focusing on methods that minimize negative performance impact, they propose Escalator (early call escalation), Engager (reordering dialog flow), and EverywhereContender (probabilistic routing) to improve cost savings and user experience without disrupting live systems.
What Problem Does This Paper Address?
Commercial spoken dialog systems (like those used for customer service) aim to reduce costs by automating phone calls. However, optimizing these systems is tricky. Simply making changes to improve efficiency can sometimes backfire, leading to worse performance and increased costs, which is unacceptable for live, customer-facing applications. The challenge is to find ways to improve these systems without risking significant negative impacts, especially during the optimization phase. The authors frame this as a "minimally invasive surgery" problem, where improvements must be safe and effective.
Methods
The paper proposes three main techniques:
- Escalator: This technique focuses on identifying calls that are unlikely to be successfully automated and escalating them to a human agent earlier in the process. This is achieved by either:
- Offline Pruning: Analyzing historical call data to identify dialog nodes or paths that consistently lead to poor outcomes or long handling times, and then removing them from the dialog flow.
- Dynamic Escalation: Using a probability estimator to predict the likelihood of a call failing at various points and escalating if a threshold is met.
- Engager: This method addresses the ordering of questions or dialog turns. By analyzing call data, it determines the optimal sequence of questions to ask to minimize the average number of turns required to gather necessary information, thereby reducing handling time.
- EverywhereContender: This is an online technique where multiple versions (contenders) of a dialog system are run simultaneously. Traffic is probabilistically routed to different contenders, and their performance is measured. The system then dynamically adjusts the routing probabilities to favor the best-performing contenders, allowing for live experimentation and optimization with minimal risk by controlling the amount of traffic sent to any single contender.
Datasets
The paper uses log data from previously deployed systems for evaluation:
- Cable Internet Troubleshooting Application: Used for evaluating the Escalator technique. The experiment involved data collected over four days in March 2010, with approximately 45,631 calls.
- Call Routing Application: Used for evaluating the Engager technique. This experiment utilized data from nearly 4 million calls (3,868,014 tokens).
- Cable Television Troubleshooting Application: Used for evaluating the EverywhereContender technique. This experiment involved data from approximately 38,004 calls, with three contender splits (one four-way and two two-way).
Key Insights and Ideas
- The "Minimally Invasive" Principle: The core idea is that optimization must be safe. Changes should either be tested offline or implemented in a way that allows for gradual rollout and minimal risk to live customer interactions.
- Reward Function: The authors define a reward function that balances automation savings (preventing human agent costs) with call duration costs. This provides a quantifiable metric for optimization.
- Escalation as a Cost-Saving Strategy: Instead of solely focusing on automating more calls, the paper highlights that intelligently escalating problematic calls early can also lead to significant cost savings by reducing overall handling time.
- Dialog Flow Reordering: The order in which information is requested can have a substantial impact on efficiency. Engager shows that optimizing this sequence can reduce the number of turns and thus handling time.
- Probabilistic Routing for Live Optimization: EverywhereContender offers a practical way to test and deploy new dialog strategies in a live environment by gradually shifting traffic to better-performing versions, minimizing the risk of widespread failure.
Key Results
- Escalator (Offline Pruning): Pruning dialog nodes led to an increase in average reward by 13.3 seconds compared to the baseline system, reaching an optimum when 176 node types were pruned.
- Engager: In a call routing application, Engager reduced the average number of questions per call from 4 to 2.87, resulting in an average reward gain of 10.5 seconds.
- EverywhereContender: In a cable television troubleshooting scenario, the EverywhereContender approach achieved an overall performance gain of 29.4 seconds compared to a baseline system using equiprobable contenders.
Limitations and Future Work
- Dynamic Escalator and EverywhereContender: The paper notes that the dynamic versions of Escalator (which predict failure during a call) and EverywhereContender (which uses dynamic routing probabilities based on real-time variables) require extensive integration with conventional dialog system architectures. These advanced versions have not yet been implemented in commercial applications due to their complexity and potential for unpredictable behavior.
- Statistical Significance: For EverywhereContender, the authors mention that with limited data (e.g., 38,000 calls), it was difficult to achieve statistical significance for most contender splits, except in one case where a contender performed significantly worse. More data is needed to reliably identify and eliminate underperforming contenders.
- Future Work: The authors suggest that further research into dynamic versions of these techniques, along with integration into more complex dialog system architectures, is a promising direction. They also allude to connections with reinforcement learning and partially observable Markov decision processes for more advanced optimization.
Summary updated