Scalable and Robust Self-Learning for Skill Routing in Large-Scale Conversational AI Systems

Mohammad Kachuee, Jinseok Nam, Sarthak Ahuja, Jin-Myung Won, Sungjin Lee

Abstract: Mohammad Kachuee, Jinseok Nam, Sarthak Ahuja, Jin-Myung Won, Sungjin Lee. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track.

2022.

Introduction

In massive conversational AI systems, "skill routing" acts as the primary traffic controller. When a user says "Play Star Wars," the system must decide whether to open a music skill, a movie skill, or a trivia game. Traditional systems have historically relied on manual rules or human-labeled data, which are impossible to scale as thousands of new skills are onboarded.

While machine learning models—specifically contextual bandits—can learn from user satisfaction signals, they are notoriously risky in production. A "pure" learning model might suddenly change its behavior for a critical command, leading to a broken user experience. The challenge addressed in this paper is: how to enable a system to learn and explore new routing options autonomously while maintaining industrial-grade robustness and stability.

Proposed Methodology

The authors propose a Hybrid Policy (HP) architecture that combines two distinct models to balance safety and optimization:

  1. Replication Policy (RP): This model is trained using standard cross-entropy loss to mimic the behavior of the current production system, acting as a "safe" baseline.
  2. Learning Policy (LP): This is a contextual bandit model trained using Inverse Propensity Scoring (IPS). Its goal is to maximize a "reward signal"—a proxy for user satisfaction derived from implicit and explicit user feedback.
Iterative self-learning cycle for updating the Hybrid Policy.

To bridge these two, the system introduces the Reference Policy Decision Rate (RPDR). This is a stochastic gatekeeper that calculates how much the LP deviates from the current system. If the deviation is too high, the RPDR stochastically decides whether to use the "safe" RP or the "experimental" LP. This ensures the overall system maintains a minimum replication rate (e.g., 90% or 99%). Robustness constraints are applied at the NLU intent level, allowing the system to be more adventurous with ambiguous queries while remaining strictly conservative with business-critical commands.

Experimental Setup

The study was conducted on Amazon Alexa, a commercial large-scale conversational system. The scale of the experiments was significant:

  • Customer Base: Online A/B experiments involved approximately 6 million unique customers.
  • Data Volume: Evaluation was performed on roughly 40 million utterances collected over a week.
  • Training Window: Models were typically trained on a sliding two-week window of traffic data.

Key Results and Insights

In online A/B tests, the self-learning models showed a consistent upward trend in rewards (user satisfaction). Even a 1% improvement in such a highly optimized system is considered a significant win.

Relative change in online reward for successive iterations of the self-learning system.

The hybrid approach successfully maintained high replication rates (97-98%) relative to the baseline, proving that the system could learn without "forgetting" how to handle standard requests. Qualitatively, the system learned to fix "mis-routings." For example, the query "what is the best seasoning for mahi-mahi" was originally routed to a Shopping skill; the self-learning system correctly rerouted it to a Knowledge/Q&A skill based on user satisfaction.

Furthermore, the authors found a high correlation ($r=0.89$) between Off-Policy Evaluation (OPE) estimates and actual online A/B performance. This allows for reliable offline guardrails before any model hits production.

Conclusion

This research demonstrates a scalable framework for continuous improvement in conversational AI. By decoupling update cycles—updating the LP daily to catch trends while updating the RP weekly—the system allows the RP to act as a "moving average filter," gradually absorbing successful behaviors while filtering out noise. Future work will focus on refining reward signal sensitivity and addressing "cold start" challenges for brand-new skills through more sophisticated exploration strategies like Thompson Sampling.

Summary updated

BibTeX
@inproceedings{kachuee2022scalable,
  author = {Mohammad Kachuee and Jinseok Nam and Sarthak Ahuja and Jin-Myung Won and Sungjin Lee},
  title = {{Scalable and Robust Self-Learning for Skill Routing in Large-Scale Conversational AI Systems}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=nlwWuy61ABk},
}
Figure