Introduction
Building and maintaining large-scale conversational AI agents like Amazon Alexa requires vast amounts of human-annotated data to train Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU) components. As these systems scale across languages and domains, manual labeling becomes a significant bottleneck. This paper, presented at IAAI-20, introduces a production-scale system that bypasses this bottleneck by allowing agents to learn directly from user interactions. By observing how users "fix" their own queries after a system failure, the system identifies successful reformulations and automatically applies them to future defective queries.
System Architecture
The proposed system operates through a dual-process architecture consisting of an online reformulation engine and an offline learning pipeline.
In the online phase, when a user provides a voice command, the Reformulation Engine intercepts the ASR output. It checks a high-performance key-value lookup table to see if the utterance has a known successful rewrite. If a rewrite exists, the reformulated query is sent to the NLU engine instead of the original, potentially defective, utterance.
In the offline phase, the system processes millions of anonymized user sessions. It groups utterances into sessions based on a 45-second inactivity window and identifies "friction" signals (e.g., the user saying "stop" or the system failing to find a result). These logs are ingested into a Markov Model that calculates the most likely successful path for any given failed interaction.
Methodology: The Absorbing Markov Chain
The core of the self-learning capability is an Absorbing Markov Chain model. To handle the massive variety of raw text, the system maps utterances to an Interpretation Space (H-space)—structured representations of intents and entities.
Interpretation and States
As shown in the figure above, raw utterances (e.g., "put on stickable me") are mapped to interpretations ($h_0, h_1, etc.$). This grouping allows the model to treat semantically similar but phonetically different requests as the same state, overcoming data sparsity.
- Transient States: These represent the various interpretations encountered during a session.
- Absorbing States: Every session eventually terminates in either Success ($r^+$) or Failure ($r^-$). Success is inferred by the absence of friction signals following an action.
Collaborative Filtering
By calculating transition probabilities across millions of users, the system uses the Fundamental Matrix of the Markov Chain to determine which "target" interpretation has the highest probability of leading to a success state from a given "source" interpretation. This approach follows a "Do No Harm" principle: if the current interpretation is already the most likely to succeed, the system will not rewrite it, ensuring high precision.
Key Results
The system was evaluated using three months of anonymized Alexa log data and a manually annotated set of over 5,000 unique utterance-rewrite pairs.
- Defect Reduction: In production A/B testing, the system reduced the defect rate by over 30% for targeted utterances.
- Win-Loss Ratio: The model achieved an 11.8 win-loss ratio, meaning it improved the user experience nearly 12 times for every one time it introduced an error.
- Accuracy: The Markov-based approach reached 93.4% accuracy, significantly outperforming standard generative baselines like Pointer-Generator networks (55.2%).
Conclusion and Future Work
This feedback-based self-learning system demonstrates that user behavior can serve as a powerful, implicit labeling mechanism for conversational AI. By leveraging an absorbing Markov Chain at scale, the system effectively automates the correction of ASR and NLU errors without manual intervention. Future research aims to address "intent shifting"—where the model might incorrectly learn a change of mind as a correction—and to refine automated blacklisting to prevent information loss during the rewriting process.
Summary updated