Abstract
A key solution to the Information Retrieval (IR)-based question answering (QA) models is to retrieve the most similar knowledge entries of a given query from a QA knowledge base, and then rerank those knowledge entries with semantic matching models. In this paper, we aim to improve an IR based e-commerce QA system—AliMe—with proposed text matching models, including a basic Triple Convolutional Neural Network (TCNN) model and two Attention-based TCNN (ATCNN) models.
Introduction
In the domain of e-commerce customer service, retrieval-based question-answering (QA) systems like Alibaba's AliMe are essential for handling high volumes of user queries. These systems typically function in two stages: first, retrieving candidate entries from a Knowledge Base (KB) using fast search engines, and second, reranking those candidates using deep learning models to find the optimal match.
Traditionally, reranking models focus on pairwise matching between a user's Query ($Q$) and a Knowledge Title ($T$), or between $Q$ and an Answer ($A$). This paper argues that such approaches miss critical context. By moving to a "triple" matching paradigm that simultaneously analyzes $Q$, $T$, and $A$, the authors demonstrate significant improvements in semantic matching accuracy.
Proposed Architectures
The core innovation is the Triple Convolutional Neural Network (TCNN), where three weight-sharing CNNs process the query, title, and answer in parallel.
Basic TCNN
The basic TCNN architecture utilizes three parallel towers to generate sentence representations for the Title, Query, and Answer. These representations are derived through initialized word embeddings, wide convolution layers, and pooling, which are then concatenated and passed to a logistic regression layer to predict relevance.
Attention-based TCNN (ATCNN)
To further enhance the model, the authors introduced attention mechanisms:
- ATCNN-1: Incorporates attention matrices ($A_{qt}$ and $A_{qa}$) to capture the interdependence between the query and the title/answer. These matrices influence the convolution and pooling layers, focusing the model on the most relevant word pairs.
- ATCNN-2 (Refined Attention): This model recognizes that the semantic relationship between a query and a title is fundamentally different from that of a query and an answer. ATCNN-2 generates separate attention feature maps for each relationship and utilizes attention-based average pooling to capture these distinct signals.
Experimental Evaluation
The models were evaluated using real-world data from Alibaba’s chatbot logs, comprising a balanced dataset of approximately 50,000 labeled pairs.
Results
ATCNN-2 emerged as the top performer, achieving an $F_1$ score of 0.928. Notably, the model maintains a production-ready inference time of approximately 1ms, making it suitable for high-traffic, real-time environments.
| Method | Precision@1 | Recall@1 | $F_1$@1 |
|---|---|---|---|
| Bi-LSTM | 0.859 | 0.975 | 0.913 |
| ABCNN-3 | 0.878 | 0.953 | 0.914 |
| TCNN | 0.892 | 0.944 | 0.918 |
| ATCNN-1 | 0.882 | 0.969 | 0.924 |
| ATCNN-2 | 0.890 | 0.969 | 0.928 |
Conclusion
The TCNN framework demonstrates that answer text is a rich source of semantic features that should be utilized during the matching phase, not just as a final output. By acknowledging the distinct linguistic roles of titles and answers through refined attention, the ATCNN-2 model provides a tangible edge over standard pairwise matching models in e-commerce QA systems.
Summary updated