TL;DR
This paper explores predicting whether a user will successfully complete a task with an automated troubleshooter (like a voice assistant). They found that n-gram models, which consider sequences of recent interactions, significantly outperform simpler linear models, especially in longer conversations, offering a path to proactively improve user experience.
What Problem Does This Paper Address?
Automated troubleshooters, like those used for phone support or voice assistants, aim to reduce costs by handling user requests without human intervention. However, when these systems fail, the call is escalated to a human agent, incurring higher costs. Identifying calls likely to fail early allows the system to either escalate proactively or adapt its strategy to prevent failure, thus saving time and reducing user frustration. Existing methods often struggle with longer, more complex dialogs, where the sheer number of turns can lead to data sparsity and reduced prediction accuracy. This paper investigates how well different modeling approaches can predict task completion in these longer, more realistic scenarios.
Methods
The authors compare two main modeling techniques for predicting task completion:
- Linear Modeling: This approach treats the entire dialog history up to the current turn as a single feature vector. Each turn's parameters are concatenated, and a model is trained for each specific turn number. This method is susceptible to data sparsity as conversations get longer.
- N-gram Modeling: This novel approach focuses on the last n turns of the conversation. By using n-grams of interaction parameters (e.g., trigrams for n=3), the model captures recent dialog context. This is hypothesized to be more robust to data sparsity in longer dialogs.
They also introduce and evaluate a set of interaction parameters derived from system logs, including ASR (Automatic Speech Recognition) status, confidence scores, user barge-ins, modality changes, and dialog manager states. Feature selection using Information Gain Ratio and a genetic algorithm is employed to identify the most relevant parameters for prediction at different points in the dialog. For classification, they use Support Vector Machines (SVMs) with a linear kernel.
Datasets
The study uses a corpus of 104,800 calls from an automated Internet troubleshooter. For their analysis, they specifically focus on a subset of 41,422 calls related to recovering lost Internet connections. To ensure balanced class representation, they create a balanced set of 4,542 calls from each of the three classes ('not_solved', 'partially_solved', 'solved'). This balanced set is further split for feature selection and training/testing using cross-validation.
Key Insights and Ideas
- Longer Dialogs are Harder: Simple linear models, which concatenate all past features, struggle significantly as dialogs extend. This is due to data sparsity – fewer calls reach later turns, making the feature vectors less reliable.
- Context is King (Especially Recent Context): N-gram models, by focusing on the immediate past (e.g., the last 3 turns), show a much more stable and higher performance in longer conversations. This suggests that recent interaction patterns are strong indicators of future success.
- Feature Importance Shifts: The relevance of different feature groups changes throughout the dialog. Early on, utterance-related features and operator requests are key. Later, as users might be more committed to the system, features related to ASR performance, barge-ins, and dialog duration become more critical.
- Enhanced Features Help: Augmenting the basic interaction parameters with running tallies, percentages, and mean values of certain features (e.g., average ASR confidence) further boosts classifier performance, even for n-gram models.
Key Results
- The n-gram modeling approach significantly outperforms linear modeling, especially at later prediction points (turns 9-25).
- While linear models show a performance drop in later turns, n-gram models remain relatively stable.
- The enhanced feature set (including running tallies, means, and percentages) consistently yields higher accuracy than the basic feature set for both linear and n-gram models.
- The paper demonstrates that it is possible to predict task completion, with accuracies generally above 33% (the baseline for a three-class problem) and significantly higher for the proposed methods.
Limitations and Future Work
- Static Models: Both linear and n-gram models are static, meaning they use a fixed feature vector for a given turn. The authors suggest that for continuous tasks, Hidden Markov Models (HMMs) might offer better performance.
- "Magic Bullet" Fallacy: The authors acknowledge that perfect prediction is unlikely, as even human raters struggle with sudden user hang-ups or opt-outs. The goal is to improve prediction, not achieve 100% accuracy.
- Generalizability: The current study is based on a single corpus. The authors plan to test the generalizability of their approaches on two more large databases from different domains.
- Binary vs. Ternary Classification: For practical deployment, reducing the problem to a binary classification (solved/not solved) or using cost-sensitive classification could improve recall for the "not_solved" class. The current classifier can provide "soft" decisions, ranking calls by risk.
Summary updated