Abstract
Minghui Qiu, Feng-Lin Li, Siyu Wang, Xing Gao, Yan Chen, Weipeng Zhao, Haiqing Chen, Jun Huang, Wei Chu. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).
2017.
Introduction
Open-domain chatbots typically rely on either Information Retrieval (IR) or Sequence-to-Sequence (Seq2Seq) generation. IR models excel at precision for common queries by searching massive databases of question-answer (QA) pairs but struggle with "long-tail" queries. Conversely, generation models can synthesize responses for any input but often produce generic or nonsensical outputs. AliMe Chat, developed by Alibaba, addresses these limitations through a hybrid engine that combines the reliability of IR with the flexibility of attentive Seq2Seq models.
System Architecture
The core of AliMe Chat is a hybrid pipeline that uses a reranking mechanism to decide between retrieved and generated answers.
The process follows four main steps:
- Candidate Retrieval: An IR model (BM25) retrieves the top 10 candidate QA pairs from a knowledge base of over 9 million pairs.
- Attentive Reranking: An attentive Seq2Seq model scores these candidates. Unlike standard generators, this model calculates a confidence score (mean probability) for each pair.
- Threshold Logic: If the top candidate's score exceeds a threshold ($T$), the system outputs the retrieved answer. If not, it triggers a generation model.
- Generation: A GRU-based Seq2Seq model with attention synthesizes a new response when no suitable match is found in the database.
Experimental Setup
The models were trained on Alibaba's internal chat logs, comprising 9,164,834 QA pairs. To focus on open-domain interaction, business-specific keywords were filtered out. Evaluation was conducted using manual labeling by business analysts on randomly sampled sets of 500 to 1,000 questions.
Key Results and Insights
The hybrid approach significantly outperforms standalone models. In online A/B testing, AliMe Chat achieved a 60.36% top-1 accuracy, compared to 40.86% for the IR-only baseline.
The "aha moment" of the research is using the reranker as a gatekeeper. By setting an empirical threshold (e.g., $T=0.19$), the system avoids the hallucinations of generation models for common questions while mitigating the "no-match" failures of IR for rare queries. Furthermore, scoring existing human-written answers (reranking) proved more effective than pure generation for maintaining naturalness.
For production readiness, the authors optimized the system to handle 42 Queries Per Second (QPS) with response times under 200ms through asynchronous ranking and parallel matrix operations.
Limitations and Future Work
Despite its success, AliMe Chat faces challenges in multi-turn context sensitivity. Current strategies for handling dialogue history are relatively simple and do not always scale well. Future work aims to incorporate "personification"—giving the chatbot a consistent character and emotional intelligence—to enhance user engagement.
Summary updated