Scalable and Robust Self-Learning for Skill Routing in Large-Scale Conversational AI Systems

Mohammad Kachuee, Jinseok Nam, Sarthak Ahuja, Jin-Myung Won, Sungjin Lee

Abstract

Skill routing is a critical component in large-scale conversational systems, determining which internal application should handle a user's request. While state-of-the-art systems have moved toward model-based approaches, providing the necessary supervision signals remains a challenge. Traditional methods like human annotation or rule replication are costly and difficult to scale. Furthermore, frequent model updates in production environments risk degrading the user experience. This paper introduces a scalable self-learning framework that utilizes contextual bandits and a unique hybrid policy to balance exploration with stability. By incorporating off-policy evaluation (OPE) as a deployment gatekeeper, the system achieves incremental improvements through frequent refreshes without the need for constant A/B testing.

Introduction

In conversational AI, "skill routing" acts as the central traffic controller. For a system like Alexa, a single utterance—such as "Play some jazz"—could potentially be handled by multiple services. The system's task is to select the optimal skill-interpretation pair to maximize user satisfaction.

Historically, this was managed through rigid, hand-crafted rules. While these rules provide a safe and predictable user experience, they fail to scale as thousands of new skills are onboarded. Transitioning to machine learning models introduces a new set of problems: human annotation is too slow for the pace of modern software deployment, and standard reinforcement learning can be volatile, potentially breaking popular features during a model update. The challenge lies in building a system that learns from user feedback automatically while ensuring it never regresses on business-critical tasks.

Methodology: The Self-Learning Framework

The researchers propose a self-learning loop based on Contextual Bandits, designed to learn directly from user interactions. The framework consists of three primary pillars:

1. Dual-Model Architecture

The system maintains two distinct models to manage the trade-off between stability and innovation:

  • Replication Policy (RP): This model is trained to mimic the current production system's behavior. It serves as a safe baseline, ensuring that the system continues to handle well-known requests reliably.
  • Learning Policy (LP): This model is trained using Inverse Propensity Scoring (IPS) to maximize a reward signal derived from user satisfaction. The LP is the "explorer," identifying better routing alternatives that the current system might miss.

2. The Hybrid Policy (HP) and RPDR

To prevent abrupt changes in the user experience, the authors introduce a Hybrid Policy (HP). This policy acts as a safety valve by calculating a Reference Policy Decision Rate (RPDR). For every incoming request, the system stochastically decides whether to follow the safe RP or the experimental LP. By setting a high RPDR (e.g., 97-99%), the system ensures that the vast majority of traffic is handled by the proven baseline, while still allowing enough exploration to gather data for the LP.

3. Off-Policy Evaluation (OPE)

Before any new model is deployed to production, it undergoes Off-Policy Evaluation. Using historical logs, the system simulates how the new model would have performed. This allows the team to apply automated "guardrails"—if the simulation predicts a performance dip in critical domains, the deployment is automatically aborted.

Model Architecture

The underlying model uses a Bi-LSTM to encode sequences of possible interpretations (hypotheses) generated by the Natural Language Understanding (NLU) unit. These encodings are processed by a Multi-Layer Perceptron (MLP) and a Softmax layer to rank and select the best skill-interpretation pair.

Experimental Setup and Datasets

The framework was tested on a massive scale using data from a commercial conversational AI system:

  • Traffic Volume: The evaluation involved approximately 40 million utterances per week.
  • User Base: Online A/B testing was conducted with roughly 6 million unique customers.
  • Update Frequency: Models were refreshed as frequently as once per week, trained on two-week windows of interaction data.

Key Insights

  • Controlled Deviation: The "aha moment" of the research is that developers do not have to choose between a static system and a volatile one. The RPDR allows for "tuned aggressiveness," where the learning rate can be adjusted per domain.
  • OPE Reliability: The study found a high correlation (r=0.89) between offline OPE simulations and actual online A/B results. This validates OPE as a robust gatekeeper, significantly reducing the reliance on expensive live experiments.
  • Automated Evolution: Because the system is safe and automated, it can absorb new skills and changing user trends much faster than human-in-the-loop systems.

Results and Impact

In live A/B testing, the self-learning models demonstrated a consistent upward trend in user satisfaction. The system achieved a ~1% improvement in average reward. While a 1% gain might seem modest, in a highly-optimized production system serving millions of users, it represents a significant enhancement in user experience and operational efficiency. Furthermore, the Hybrid Policy successfully maintained a 97-98% similarity to the baseline, proving that the system could learn without forgetting critical legacy behaviors.

Conclusion and Future Work

The proposed framework demonstrates that self-learning in large-scale AI is possible without sacrificing robustness. However, challenges remain. Future research will address the "cold start" problem for new domains where no baseline policy exists, as well as potential estimation biases in OPE caused by weight clipping. Additionally, exploring more granular reward signals beyond general user satisfaction could further refine the routing logic.

Summary updated

Abstract

Skill routing is an important component in large-scale conversational systems. In contrast to traditional rule-based skill routing, state-of-the-art systems use a model-based approach to enable natural conversations. To provide supervision signal required to train such models, ideas such as human annotation, replication of a rule-based system, relabeling based on user paraphrases, and bandit-based learning were suggested. However, these approaches: (a) do not scale in terms of the number of skills and skill on-boarding, (b) require a very costly expert annotation/rule-design, (c) introduce risks in the user experience with each model update. In this paper, we present a scalable self-learning approach to explore routing alternatives without causing abrupt policy changes that break the user experience, learn from the user interaction, and incrementally improve the routing via frequent model refreshes. To enable such robust frequent model updates, we suggest a simple and effective approach that ensures controlled policy updates for individual domains, followed by an off-policy evaluation for making deployment decisions without any need for lengthy A/B experimentation. We conduct various offline and online A/B experiments on a commercial large-scale conversational system to demonstrate the effectiveness of the proposed method in real-world production settings.
BibTeX
@inproceedings{kachuee2022scalable,
  author = {Mohammad Kachuee and Jinseok Nam and Sarthak Ahuja and Jin-Myung Won and Sungjin Lee},
  title = {{Scalable and Robust Self-Learning for Skill Routing in Large-Scale Conversational AI Systems}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=nlwWuy61ABk},
}
Figure