Multi-task Sentence Encoding Model for Semantic Retrieval in Question Answering Systems

Qiang Huang, Jianhui Bu, Weijian Xie, Shengwen Yang, Weijia Wu, Liping Liu

TL;DR

Researchers at Baidu have developed the Multi-task Sentence Encoding Model (MSEM), which significantly improves how Question Answering (QA) systems find similar questions in massive databases. By training a model to simultaneously identify paraphrases and classify the "intent" of a sentence, and then pairing it with Approximate Nearest Neighbor (ANN) search, they achieved state-of-the-art performance and a 14% F1-score boost in real-world online deployment.

What Problem Does This Paper Address?

Most Information Retrieval (IR)-based QA systems (like those powering Alexa or Siri) follow a two-step process: first, they use keyword-based search (like Lucene) to grab a few dozen candidates, and then they use a complex "re-ranking" model to find the best match.

The authors identify two major flaws in this approach:

  1. The Re-ranking Bottleneck: Because re-ranking models are computationally expensive, they can only look at a tiny fraction of the database. If the keyword search misses the right question, the system fails.
  2. Ignoring Transitivity: Traditional models treat every pair of sentences as independent. They ignore the fact that if Question A is a paraphrase of Question B, and B is a paraphrase of C, then A and C must also be paraphrases.

Methods

The authors propose MSEM, an encoding-based model designed for speed and semantic depth.

1. Multi-task Learning

Instead of just learning "Are these two sentences the same?", the model is trained on two tasks at once:

  • Paraphrase Identification (PI): A binary task to see if two questions match.
  • Intent Classification: The authors built a graph where paraphrased questions are connected. Each connected "cluster" represents a specific intent. The model learns to classify which cluster a sentence belongs to, which forces it to learn a more robust semantic representation.

2. The Sentence Encoder Architecture

The encoder uses several sophisticated layers to turn a sentence into a fixed-length vector:

  • BiGRU: To capture sequential word information.
  • Attention Recurrent Unit (ARU): A novel component that uses position-wise multi-head attention. Unlike standard RNNs, it can be computed in parallel on a GPU, making it much faster.
  • Attentive Pooling: Instead of just averaging word vectors, the model learns which words are most important to the sentence's overall meaning.

3. Semantic Retrieval Framework

Because MSEM is an "encoding-based" model (it turns a sentence into a single vector), the authors can pre-calculate vectors for every question in their database. They use Approximate Nearest Neighbor (ANN) technology (specifically the Annoy library) to search the entire database in milliseconds, bypassing the need for old-school keyword search.

Datasets

The model was evaluated on four major datasets:

  • Quora Question Pairs: A standard English benchmark for paraphrase detection.
  • LCQMC: A large-scale Chinese question-matching corpus from Baidu Knows.
  • BQ Dataset: A Chinese dataset focused on the banking domain.
  • TCS Dataset: A real-world Chinese dataset from telephone customer service logs, featuring noisy text from speech-to-text conversion.

Key Insights and Ideas

  • The Power of Intent: Training on intent classification acts as a powerful regularizer. It helps the model understand that "How do I build a PC?" and "Steps to assemble a computer" belong to the same semantic "neighborhood," even if they share few words.
  • Encoding vs. Interaction: While "interaction-based" models (which compare words across sentences) are often more accurate, they are too slow for large-scale retrieval. MSEM proves that a well-designed encoding model can nearly match their accuracy while being orders of magnitude faster.
  • Parallelism Matters: The introduction of the ARU allows the model to leverage GPU acceleration more effectively than traditional LSTMs or GRUs, which are inherently sequential.

Key Results

  • State-of-the-Art: MSEM outperformed existing encoding-based models like SSE (Shortcut-Stacked Sentence Encoder) across all four datasets.
  • Quora Performance: On the Quora dataset, MSEM achieved 88.86% accuracy, outperforming the popular BiMPM model.
  • Real-world Impact: In an online evaluation with a telephone customer service system, the MSEM-based framework achieved an F1-score of 84.87%, a massive 14.26% improvement over the baseline keyword-based system.
  • Ablation Success: Removing the multi-task learning component dropped accuracy by about 0.75%, proving that the intent classification task is vital to the model's success.

Limitations and Future Work

While the paper demonstrates significant gains, a few areas remain for exploration:

  • Cold Start for Intents: The intent classification relies on pre-existing clusters. It is unclear how the model handles entirely new types of questions that don't fit into existing training clusters.
  • Graph Construction: Building the connected graph for intent classification requires high-quality paraphrase data; in very niche domains, this data might be hard to acquire.
  • Future Directions: The authors suggest that further refinements to the ARU and exploring different ANN indexing strategies could further reduce latency for even larger knowledge bases.

Summary updated

Abstract

Question Answering (QA) systems are used to provide proper responses to users' questions automatically. Sentence matching is an essential task in the QA systems and is usually reformulated as a Paraphrase Identification (PI) problem. Given a question, the aim of the task is to find the most similar question from a QA knowledge base. In this paper, we propose a Multi-task Sentence Encoding Model (MSEM) for the PI problem, wherein a connected graph is employed to depict the relation between sentences, and a multi-task learning model is applied to address both the sentence matching and sentence intent classification problem. In addition, we implement a general semantic retrieval framework that combines our proposed model and the Approximate Nearest Neighbor (ANN) technology, which enables us to find the most similar question from all available candidates very quickly during online serving. The experiments show the superiority of our proposed method as compared with the existing sentence matching models.
BibTeX
@inproceedings{huang2019multi,
  author = {Qiang Huang and Jianhui Bu and Weijian Xie and Shengwen Yang and Weijia Wu and Liping Liu},
  title = {{Multi-task Sentence Encoding Model for Semantic Retrieval in Question Answering Systems}},
  booktitle = {IJCNN},
  year = {2019},
  url = {https://openreview.net/forum?id=9eL87y4edAf},
}

Loading related content...

Figure