Empower Large Language Model to Perform Better on Industrial Domain-Specific Question Answering

Zezhong Wang, Fangkai Yang, Pu Zhao, Lu Wang, Jue Zhang, Mohit Garg, Qingwei Lin, Dongmei Zhang

TL;DR

This paper tackles the challenge of making Large Language Models (LLMs) perform well on specialized industrial questions by introducing a new dataset (MSQA) and a novel "model interaction paradigm." This approach empowers LLMs with domain-specific knowledge, leading to significantly better question-answering performance compared to traditional retrieval methods.

What Problem Does This Paper Address?

LLMs excel at general knowledge but struggle with specific, technical questions common in industrial settings due to a lack of domain-specific training data. Fine-tuning LLMs for every domain is expensive and risks data leakage. Existing methods like retrieval-augmented generation can suffer from incomplete or out-of-context information. This paper aims to bridge this gap by providing a way for LLMs to effectively leverage domain-specific knowledge for industrial question answering, particularly for long-form answers.

Methods

The authors propose a two-step "model interaction paradigm":

  1. Obtain a Domain-Specific Model: They pre-train smaller language models (like LLaMA) on extensive Azure documentation. This model is then fine-tuned for the Long-Form Question Answering (LFQA) task using instruction tuning. This creates a specialized model that "understands" the domain.
  2. Empower the LLM: At runtime, this domain-specific model acts as an "expert," providing relevant domain knowledge to a larger, general-purpose LLM (like GPT-4). The LLM then uses both the original question and this provided domain knowledge to generate a more accurate and contextually rich answer. This approach avoids directly replacing the LLM but augments its capabilities.

Datasets

  • MSQA Dataset: The authors introduce a new benchmark dataset called MSQA, specifically for cloud-domain question answering. It comprises 32,000 QA pairs sourced from the Microsoft Q&A forum, focusing on Microsoft products and IT technical issues. This dataset is unique in its focus on industry cloud-specific QA knowledge.
  • Azure Documentation: Used for pre-training the domain-specific models.

Key Insights and Ideas

  • Domain-Specific Models as "Experts": The core idea is to use a smaller, specialized model as a knowledge provider for a larger, general LLM. This is more efficient and manageable than fine-tuning the large LLM itself.
  • Bridging General and Specific Knowledge: The interaction paradigm effectively combines the broad understanding of general LLMs with the deep, nuanced knowledge of domain-specific models.
  • Beyond Retrieval: The proposed method aims to overcome the limitations of retrieval-based approaches, such as the "quote out of context" problem, by generating domain knowledge rather than just retrieving text snippets.
  • Importance of Domain-Specific Benchmarks: The creation of the MSQA dataset highlights the need for specialized benchmarks to accurately evaluate LLM performance in real-world industrial scenarios.

Key Results

  • The proposed LLM+EXP approach (using the domain-specific model) significantly outperformed baseline methods (Raw LLM, LLM+BM25, LLM+DPR) across various metrics, including semantic overlap (BERT-Score, SIM) and custom metrics like Keyword/Span-Hit-Rate (KHR) and Can-Answer-Rate (CAR).
  • For example, with GPT-4 as the backbone, LLM+EXP achieved a BERT-Score of 56.91 and a SIM score of 71.19, notably higher than the raw LLM.
  • Human evaluations and LLM-based evaluations also favored the LLM+EXP method, indicating more accurate and relevant answers.
  • The Can-Answer-Rate (CAR) was significantly higher for LLM+EXP, suggesting fewer "hallucinated" or unconfident answers.

Limitations and Future Work

  • Dataset Specificity: The MSQA dataset and the domain-specific model are currently limited to the Microsoft Azure cloud domain. This may impact the generalizability of the approach to other industrial domains.
  • Parameter Tuning: Instruction tuning, while effective, presents challenges in determining optimal parameters like the number of epochs.
  • Automated Metrics: Developing robust, automated metrics for evaluating long-form answers remains an ongoing challenge.
  • Multimodality: The current work focuses on text-based QA, excluding scenarios involving image attachments.
  • Language Support: The model is trained and evaluated exclusively in English, while the source Q&A forum contains multiple languages.

Future work could involve expanding the dataset to cover more domains, exploring more advanced parameter tuning strategies, developing better evaluation metrics, and incorporating multimodal capabilities.

Summary updated

Abstract

Large Language Model (LLM) has gained popularity and achieved remarkable results in open-domain tasks, but its performance in real industrial domain-specific scenarios is average since there is no specific knowledge in it. This issue has attracted widespread attention, but there are few relevant benchmarks available. In this paper, we provide a benchmark Question Answering (QA) dataset named MSQA, which is about Microsoft products and IT technical problems encountered by customers. This dataset contains industry cloud-specific QA knowledge, which is not available for general LLM, so it is well suited for evaluating methods aimed at improving domain-specific capabilities of LLM. In addition, we propose a new model interaction paradigm that can empower LLM to achieve better performance on domain-specific tasks where it is not proficient. Extensive experiments demonstrate that the approach following our model fusion framework outperforms the commonly used LLM with retrieval methods.
BibTeX
@inproceedings{wang2023empower,
  author = {Zezhong Wang and Fangkai Yang and Pu Zhao and Lu Wang and Jue Zhang and Mohit Garg and Qingwei Lin and Dongmei Zhang},
  title = {{Empower Large Language Model to Perform Better on Industrial Domain-Specific Question Answering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-industry.29},
}
Figure