Feedback-Based Self-Learning in Large-Scale Conversational AI Agents

Pragaash Ponnusamy, Alireza Roshan Ghias, Chenlei Guo, Ruhi Sarikaya

Introduction

Building and maintaining large-scale conversational AI agents like Amazon Alexa requires vast amounts of human-annotated data to train Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU) components. As these systems scale across languages and domains, manual labeling becomes a significant bottleneck. This paper, presented at IAAI-20, introduces a production-scale system that bypasses this bottleneck by allowing agents to learn directly from user interactions. By observing how users "fix" their own queries after a system failure, the system identifies successful reformulations and automatically applies them to future defective queries.

System Architecture

The proposed system operates through a dual-process architecture consisting of an online reformulation engine and an offline learning pipeline.

The system architecture distinguishes between the real-time online reformulation engine and the offline daily update cycle that processes anonymized logs.

In the online phase, when a user provides a voice command, the Reformulation Engine intercepts the ASR output. It checks a high-performance key-value lookup table to see if the utterance has a known successful rewrite. If a rewrite exists, the reformulated query is sent to the NLU engine instead of the original, potentially defective, utterance.

In the offline phase, the system processes millions of anonymized user sessions. It groups utterances into sessions based on a 45-second inactivity window and identifies "friction" signals (e.g., the user saying "stop" or the system failing to find a result). These logs are ingested into a Markov Model that calculates the most likely successful path for any given failed interaction.

Methodology: The Absorbing Markov Chain

The core of the self-learning capability is an Absorbing Markov Chain model. To handle the massive variety of raw text, the system maps utterances to an Interpretation Space (H-space)—structured representations of intents and entities.

Illustration of (a-c) user sessions, (d) the transition probabilities in the Markov Chain, and (e) the mapping of raw utterances to the semantic H-space.

Interpretation and States

As shown in the figure above, raw utterances (e.g., "put on stickable me") are mapped to interpretations ($h_0, h_1, etc.$). This grouping allows the model to treat semantically similar but phonetically different requests as the same state, overcoming data sparsity.

  • Transient States: These represent the various interpretations encountered during a session.
  • Absorbing States: Every session eventually terminates in either Success ($r^+$) or Failure ($r^-$). Success is inferred by the absence of friction signals following an action.

Collaborative Filtering

By calculating transition probabilities across millions of users, the system uses the Fundamental Matrix of the Markov Chain to determine which "target" interpretation has the highest probability of leading to a success state from a given "source" interpretation. This approach follows a "Do No Harm" principle: if the current interpretation is already the most likely to succeed, the system will not rewrite it, ensuring high precision.

Key Results

The system was evaluated using three months of anonymized Alexa log data and a manually annotated set of over 5,000 unique utterance-rewrite pairs.

  • Defect Reduction: In production A/B testing, the system reduced the defect rate by over 30% for targeted utterances.
  • Win-Loss Ratio: The model achieved an 11.8 win-loss ratio, meaning it improved the user experience nearly 12 times for every one time it introduced an error.
  • Accuracy: The Markov-based approach reached 93.4% accuracy, significantly outperforming standard generative baselines like Pointer-Generator networks (55.2%).

Conclusion and Future Work

This feedback-based self-learning system demonstrates that user behavior can serve as a powerful, implicit labeling mechanism for conversational AI. By leveraging an absorbing Markov Chain at scale, the system effectively automates the correction of ASR and NLU errors without manual intervention. Future research aims to address "intent shifting"—where the model might incorrectly learn a change of mind as a correction—and to refine automated blacklisting to prevent information loss during the rewriting process.

Summary updated

Abstract

Today, most of the large-scale conversational AI agents such as Alexa, Siri, or Google Assistant are built using manually annotated data to train the different components of the system including Automatic Speech Recognition (ASR), Natural Language Understanding (NLU) and Entity Resolution (ER). Typically, the accuracy of the machine learning models in these components are improved by manually transcribing and annotating data. As the scope of these systems increase to cover more scenarios and domains, manual annotation to improve the accuracy of these components becomes prohibitively costly and time consuming. In this paper, we propose a system that leverages customer/system interaction feedback signals to automate learning without any manual annotation. Users of these systems tend to modify a previous query in hopes of fixing an error in the previous turn to get the right results. These reformulations, which are often preceded by defective experiences caused by either errors in ASR, NLU, ER or the application. In some cases, users may not properly formulate their requests (e.g. providing partial title of a song), but gleaning across a wider pool of users and sessions reveals the underlying recurrent patterns. Our proposed self-learning system automatically detects the errors, generate reformulations and deploys fixes to the runtime system to correct different types of errors occurring in different components of the system. In particular, we propose leveraging an absorbing Markov Chain model as a collaborative filtering mechanism in a novel attempt to mine these patterns. We show that our approach is highly scalable, and able to learn reformulations that reduce Alexa-user errors by pooling anonymized data across millions of customers. The proposed self-learning system achieves a win-loss ratio of 11.8 and effectively reduces the defect rate by more than 30% on utterance level reformulations in our production A/B tests. To the best of our knowledge, this is the first self-learning large-scale conversational AI system in production.
BibTeX
@inproceedings{ponnusamy2020feedback,
  author = {Pragaash Ponnusamy and Alireza Roshan Ghias and Chenlei Guo and Ruhi Sarikaya},
  title = {{Feedback-Based Self-Learning in Large-Scale Conversational AI Agents}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i08.7022},
}
Figure