A Technical Question Answering System with Transfer Learning

Wenhao Yu, Lingfei Wu, Yu Deng, Ruchi Mahindru, Qingkai Zeng, Sinem Güven, Meng Jiang

TL;DR

This paper introduces TransTQA, a novel system that automatically answers technical questions by retrieving similar, previously answered questions. It leverages a siamese ALBERT architecture and a three-step transfer learning approach to achieve efficient and accurate responses across multiple technical domains.

What Problem Does This Paper Address?

Technical communities, like StackOverflow and company-specific forums, are invaluable for knowledge sharing. However, getting timely and accurate answers from human experts can be expensive and time-consuming. Many questions go unanswered, or the answers are buried in lengthy, asynchronous discussions. This paper addresses the challenge of automatically providing relevant answers to technical questions, especially when domain-specific knowledge is scarce or questions are complex. The authors propose a retrieval-based approach that capitalizes on the vast amount of historical Q&A data available in technical forums.

Methods

The core of TransTQA is a siamese ALBERT network. ALBERT, a more parameter-efficient variant of BERT, is used as the backbone. The siamese architecture involves two separate ALBERT encoders: one for the input question and one for a candidate answer. These encoders generate embeddings for the question and answer, which are then compared using a cosine similarity measure to determine relevance.

The system employs a three-step transfer learning strategy:

  1. Fine-tuning on Source Corpus: The pre-trained ALBERT model is first fine-tuned on a large, unstructured technical corpus (e.g., IBM Technotes). This step helps the model learn technical terminology and domain-specific language that might be absent in general pre-training data. This is done using the Masked Language Model (MLM) task.
  2. Fine-tuning on Source QA: The model is then further fine-tuned on a source technical QA dataset (e.g., AskUbuntu). This step trains the model to understand the relationship between questions and answers in a QA context.
  3. Fine-tuning on Target QA: Finally, the model is fine-tuned on the target technical QA dataset (e.g., TECHQA, StackUnix). This final step adapts the model to the specific domain of the questions it will be answering.

For optimization, TransTQA uses a multiple negatives ranking loss. This approach treats all other responses in a training batch as negative examples for a given question, encouraging the model to distinguish correct answers from incorrect ones.

Datasets

The paper mentions experiments conducted on three technical datasets:

  • TechQA (Castelli et al., 2020)
  • StackUnix
  • An IBM internal dataset

The authors also refer to AskUbuntu and StackOverflow as sources for training data, particularly for the source QA fine-tuning step.

Key Insights and Ideas

  • Transfer Learning is Crucial for Technical QA: The paper strongly emphasizes that transferring knowledge from both general technical corpora (like Technotes) and existing technical QA datasets (like AskUbuntu) significantly improves performance on new technical QA tasks.
  • Siamese ALBERT for Efficiency: Using a siamese ALBERT architecture, rather than a standard BERT approach for QA pairs, leads to substantial improvements in inference time and memory efficiency. This is because answer embeddings can be pre-computed, allowing for rapid retrieval.
  • Beyond Lexical Similarity: The authors highlight that in technical domains, simple lexical similarity might not be enough. The use of Transformer-based models like ALBERT with attention mechanisms is key to capturing complex semantic relationships between questions and answers.
  • Practical Relevance of Retrieval: The paper argues for the importance of retrieval-based QA in real-world scenarios, where users often have access to a large history of questions and answers, rather than a small set of pre-defined candidates.

Key Results

  • Superior Performance: TransTQA, especially when using knowledge transfer from both source corpora and source QA (Method M4), outperforms traditional LSTM-based QA systems and achieves the best results on the TechQA and StackUnix datasets. For instance, on TechQA, M4 with ALBERT achieved an MRR of 38.69, R@1 of 29.30, R@5 of 46.24, and R@10 of 55.20.
  • Efficiency Gains: The siamese-ALBERT model in TransTQA is significantly faster and uses less memory compared to other BERT-based retrieval QA systems. It boasts an inference time of 0.13s and 10M parameters, compared to BERT-Rerank (62.67s, 170M parameters) and BERT-Serini (0.55s, 170M parameters).
  • Case Study Success: In a case study, the system successfully identified the correct answer for a user query using the M4 transfer learning setting.

Limitations and Future Work

  • Domain Specificity: While the system aims for multi-domain support, the authors acknowledge that directly applying ALBERT to technical domains can be limited due to the lack of technical terminology in its general pre-training. The fine-tuning steps are designed to mitigate this.
  • Data Scarcity: The paper notes that emerging technical forums often have limited QA pairs, which can be a challenge for fine-tuning. The proposed transfer learning approach helps address this data shortage.
  • Future Work: The paper doesn't explicitly detail future work, but the success of the transfer learning approach suggests potential for exploring more sophisticated transfer learning techniques and applying the system to an even wider range of technical domains. The system's modular design also allows for easy integration of different ranking services.

Summary updated

Abstract

In recent years, the need for community technical question-answering sites has increased significantly. However, it is often expensive for human experts to provide timely and helpful responses on those forums. We develop TransTQA, which is a novel system that offers automatic responses by retrieving proper answers based on correctly answered similar questions in the past. TransTQA is built upon a siamese ALBERT network, which enables it to respond quickly and accurately. Furthermore, TransTQA adopts a standard deep transfer learning strategy to improve its capability of supporting multiple technical domains.
BibTeX
@inproceedings{yu2020technical,
  author = {Wenhao Yu and Lingfei Wu and Yu Deng and Ruchi Mahindru and Qingkai Zeng and Sinem Güven and Meng Jiang},
  title = {{A Technical Question Answering System with Transfer Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-demos.13},
}
Figure