TCNN: Triple Convolutional Neural Network Models for Retrieval-based Question Answering System in E-commerce

Shuangyong Song, Chao WangWWW 2020 · arXiv · OpenReview

Abstract

A key solution to the Information Retrieval (IR)-based question answering (QA) models is to retrieve the most similar knowledge entries of a given query from a QA knowledge base, and then rerank those knowledge entries with semantic matching models. In this paper, we aim to improve an IR based e-commerce QA system—AliMe—with proposed text matching models, including a basic Triple Convolutional Neural Network (TCNN) model and two Attention-based TCNN (ATCNN) models.

Introduction

In the domain of e-commerce customer service, retrieval-based question-answering (QA) systems like Alibaba's AliMe are essential for handling high volumes of user queries. These systems typically function in two stages: first, retrieving candidate entries from a Knowledge Base (KB) using fast search engines, and second, reranking those candidates using deep learning models to find the optimal match.

Traditionally, reranking models focus on pairwise matching between a user's Query ($Q$) and a Knowledge Title ($T$), or between $Q$ and an Answer ($A$). This paper argues that such approaches miss critical context. By moving to a "triple" matching paradigm that simultaneously analyzes $Q$, $T$, and $A$, the authors demonstrate significant improvements in semantic matching accuracy.

Proposed Architectures

The core innovation is the Triple Convolutional Neural Network (TCNN), where three weight-sharing CNNs process the query, title, and answer in parallel.

Basic TCNN

The basic TCNN architecture utilizes three parallel towers to generate sentence representations for the Title, Query, and Answer. These representations are derived through initialized word embeddings, wide convolution layers, and pooling, which are then concatenated and passed to a logistic regression layer to predict relevance.

The basic TCNN architecture showing the triple-tower structure for processing Title (T), Query (Q), and Answer (A).

Attention-based TCNN (ATCNN)

To further enhance the model, the authors introduced attention mechanisms:

  • ATCNN-1: Incorporates attention matrices ($A_{qt}$ and $A_{qa}$) to capture the interdependence between the query and the title/answer. These matrices influence the convolution and pooling layers, focusing the model on the most relevant word pairs.
  • ATCNN-2 (Refined Attention): This model recognizes that the semantic relationship between a query and a title is fundamentally different from that of a query and an answer. ATCNN-2 generates separate attention feature maps for each relationship and utilizes attention-based average pooling to capture these distinct signals.
The ATCNN-2 architecture, which utilizes refined attention feature maps and attention-based pooling to distinguish between Q-T and Q-A relationships.

Experimental Evaluation

The models were evaluated using real-world data from Alibaba’s chatbot logs, comprising a balanced dataset of approximately 50,000 labeled pairs.

Results

ATCNN-2 emerged as the top performer, achieving an $F_1$ score of 0.928. Notably, the model maintains a production-ready inference time of approximately 1ms, making it suitable for high-traffic, real-time environments.

Method Precision@1 Recall@1 $F_1$@1
Bi-LSTM 0.859 0.975 0.913
ABCNN-3 0.878 0.953 0.914
TCNN 0.892 0.944 0.918
ATCNN-1 0.882 0.969 0.924
ATCNN-2 0.890 0.969 0.928

Conclusion

The TCNN framework demonstrates that answer text is a rich source of semantic features that should be utilized during the matching phase, not just as a final output. By acknowledging the distinct linguistic roles of titles and answers through refined attention, the ATCNN-2 model provides a tangible edge over standard pairwise matching models in e-commerce QA systems.

Summary updated

Abstract

Automatic question-answering (QA) systems have boomed during last few years, and commonly used techniques can be roughly categorized into Information Retrieval (IR)-based and generation-based. A key solution to the IR based models is to retrieve the most similar knowledge entries of a given query from a QA knowledge base, and then rerank those knowledge entries with semantic matching models. In this paper, we aim to improve an IR based e-commerce QA system-AliMe with proposed text matching models, including a basic Triple Convolutional Neural Network (TCNN) model and two Attention-based TCNN (ATCNN) models. Experimental results show their effect.
BibTeX
@inproceedings{song2020tcnn,
  author = {Shuangyong Song and Chao Wang},
  title = {{TCNN: Triple Convolutional Neural Network Models for Retrieval-based Question Answering System in E-commerce}},
  booktitle = {The Web Conference},
  year = {2020},
  url = {https://openreview.net/forum?id=_SUEcv04Foy},
}
Figure