Empower Large Language Model to Perform Better on Industrial Domain-Specific Question Answering

Fangkai Yang, Pu Zhao, Zezhong Wang, Lu Wang, Jue Zhang, Mohit Garg, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang

TL;DR

While Large Language Models (LLMs) excel at general knowledge, they often stumble on complex, industry-specific technical questions. This paper introduces MSQA, a new 32k-pair dataset for cloud-domain question answering, and proposes a novel interaction paradigm that replaces traditional document retrieval with a smaller, domain-tuned "expert" language model to feed precise knowledge directly to a backbone LLM.

What Problem Does This Paper Address?

When engineers or customers face highly specific technical issues—such as configuring hybrid cloud environments or debugging enterprise software—general-purpose LLMs like GPT-3.5 often fail or hallucinate.

To fix this, practitioners typically turn to Retrieval-Augmented Generation (RAG), which searches a documentation database for text chunks containing matching keywords and stuffs them into the LLM's prompt. However, the authors point out two major flaws with standard RAG in industrial settings:

  1. "Quote out of Context": Retrieval algorithms look for keyword overlap but don't understand the underlying question. They often fetch text chunks that contain the right words but completely miss the actual solution logic.
  2. Prompt Length Constraints: Stuffer-heavy retrieval results can easily overwhelm the context window of an LLM, making it difficult to synthesize a clean, long-form answer.
Figure 1: An example1from MSQA dataset shows retrieval-based methods’ limitations in complex question handling. The retrieved chunks contain matching keywords (highlighted in green) but failed to retrieve essential information needed to answer the question correctly. Our model generates more accurate answers by understanding the question and leveraging domain-specific knowledge (highlighted in yellow). Case details are in Table 17 in Appendix I.

Methods

Instead of relying on traditional search engines (like BM25 or Dense Passage Retrieval), the authors propose a two-step Model Interaction Framework:

  1. Offline Training (The Expert Plugin): They take a smaller, open-source model (LLaMA-7B) and pre-train it directly on specialized technical documentation (specifically, Microsoft Azure docs). They then perform instruction tuning on domain-specific Q&A pairs so the model learns how to act as a targeted knowledge generator.
  2. Online Generation (The Interaction): At runtime, when a user asks a question, the specialized smaller model acts as an "expert." Instead of pulling raw paragraphs from a database, it generates a concise summary of the relevant domain knowledge. This generated knowledge, along with the original question, is then passed to a powerful backbone LLM (like GPT-4) to write the final, polished response.
The proposed two-step model interaction framework, showing offline expert training and online knowledge generation.

Datasets

The authors introduce MSQA (Microsoft Question Answering), a benchmark dataset specifically designed for long-form question answering (LFQA) in the cloud domain.

  • Size & Source: 32,252 QA pairs scraped and cleaned from the public Microsoft Q&A forum (spanning October 2019 to May 2023).
  • Content: Real-world technical problems categorized by 332 distinct topic tags (e.g., Azure Virtual Machine, Windows Server, M365).
  • Processing: The raw data was heavily filtered to remove noisy human artifacts like usernames, platform-generated text ("please upvote if helpful"), and broken formatting.

Key Insights and Ideas

  • Generative Knowledge > Raw Text Chunks: A fine-tuned smaller model acts as a highly effective compressor of documentation. It provides the backbone LLM with cleaner, more direct context than raw text snippets, eliminating the "quote out of context" problem.
  • The "Blind Confidence" of GPT-3.5: The authors measured a metric called Can-Answer-Rate (CAR)—the frequency with which a model feels confident enough to answer rather than saying "I don't know." When given raw retrieval chunks, GPT-3.5's confidence actually decreased, suggesting it became confused by irrelevant text. Conversely, it often exhibited blind confidence when it didn't have enough data, leading to hallucinations. GPT-4 proved much more robust and conservative.

Key Results

The authors evaluated their framework (LLM+EXP) against raw LLMs and standard RAG setups (LLM+BM25 and LLM+DPR) using semantic metrics (BERT-Score, Embedding Similarity) and human evaluation.

  • Semantic Accuracy: Using GPT-4 as the backbone, the proposed expert method (LLM+EXP) achieved the highest BERT-Score (56.91) and Semantic Similarity (71.19), outperforming the best retrieval baseline (LLM+DPR, which scored 54.83 and 68.78 respectively).
  • Human Preference: In a blind study with five domain experts ranking the generated answers, the proposed method was overwhelmingly favored:
  • LLM+EXP (Ours): Chosen as the best answer 76.67% of the time.
  • LLM+DPR: Chosen only 20.00% of the time.
  • Raw LLM: Chosen only 13.33% of the time.

Limitations and Future Work

While promising, the authors note several constraints:

  • Scope: The evaluation was strictly confined to Microsoft Azure documentation and the English language. Its generalizability to other highly technical fields (like medicine or law) remains to be proven.
  • Text-Only: The current pipeline discards images and screenshots, which are incredibly common in real-world IT troubleshooting. Moving to a multimodal expert model is a clear next step.
  • Optimization Hurdles: Setting the exact number of epochs during the instruction-tuning phase of the smaller expert model remains difficult due to a lack of automated metrics for long-form text evaluation.

Summary updated

Abstract

Large Language Model (LLM) has gained popularity and achieved remarkable results in open-domain tasks, but its performance in real industrial domain-specific scenarios is average due to its lack of specific domain knowledge. This issue has attracted widespread attention, but there are few relevant benchmarks available. In this paper, we provide a benchmark Question Answering (QA) dataset named MSQA, centered around Microsoft products and IT technical problems encountered by customers. This dataset contains industry cloud-specific QA knowledge, an area not extensively covered in general LLMs, making it well-suited for evaluating methods aiming to enhance LLMs' domain-specific capabilities. In addition, we propose a new model interaction paradigm that can empower LLM to achieve better performance on domain-specific tasks where it is not proficient. Extensive experiments demonstrate that the approach following our method outperforms the commonly used LLM with retrieval methods. We make our source code and sample data available at: https://aka.ms/Microsoft_QA.
BibTeX
@inproceedings{yang2023empower,
  author = {Fangkai Yang and Pu Zhao and Zezhong Wang and Lu Wang and Jue Zhang and Mohit Garg and Qingwei Lin and Saravan Rajmohan and Dongmei Zhang},
  title = {{Empower Large Language Model to Perform Better on Industrial Domain-Specific Question Answering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-industry.29},
}
Figure