Abstract
Instructional Question Answering (QA) over electronic device manuals presents unique challenges, including non-contiguous answers, long-range dependencies, and domain-specific terminology. This paper introduces a massive domain-specific corpus and the EMQAP (E-Manual Question Answering Pipeline) framework. By utilizing multi-task learning and domain-specific pre-training, EMQAP achieves a 40% improvement in ROUGE-L F1 scores over state-of-the-art general-purpose models.
Introduction
While open-domain "factoid" QA has seen massive progress, instructional QA—answering "how-to" queries from technical manuals—remains a significant challenge. Electronic manuals (E-manuals) are uniquely difficult for AI because answers are often scattered across sections, instructions may depend on context established paragraphs earlier, and technical terms like "port" or "address" have specific meanings distinct from general literature. Previously, the field lacked a large-scale, standardized dataset to train models specifically for this instructional domain.
Methodology: The EMQAP Framework
The authors propose a two-stage pipeline designed to handle the complexities of technical documentation.
1. Domain-Specific Pre-training
A standard RoBERTa model was further trained on a massive new corpus of over 300,000 E-manuals. To prevent the model from losing its general language knowledge (catastrophic forgetting), the researchers employed Elastic Weight Consolidation (EWC) and Learning Rate Decay (LRD). This ensures the model becomes a technical specialist while retaining basic linguistic "common sense."
2. Multi-Task Learning (MTL) Pipeline
The inference process involves unsupervised information retrieval to narrow the manual down to the top 10 relevant sections. This is followed by a supervised MTL framework that simultaneously performs two tasks: Section Retrieval (SR) (identifying the correct section) and Answer Retrieval (AR) (extracting the specific answer span).
Datasets and Evaluation
The study introduces several key resources to the research community:
- E-Manual Corpus: A pre-training set containing 307,957 manuals (~11GB of text).
- S10 QA & Smart TV QA: Expert-annotated datasets (904 and 950 pairs, respectively) based on Samsung device manuals.
- CQ-AQ Dataset: 1,028 real-world user questions pulled from Amazon Community QA forums, mapped back to relevant manual sections.
Results and Semantic Insights
EMQAP significantly outperformed baselines like DPR and TAP, doubling Exact Match (EM) scores on the S10 dataset. A key finding was the dramatic shift in the model's internal semantic representation after domain-specific training.
As shown in the table above, words like "port," "switch," or "key" moved from general associations to tech-specific ones (e.g., "ip," "server," "ethernet"). This specialization allows the model to accurately parse procedural instructions that general-purpose models might misinterpret.
Conclusion
The EMQAP framework demonstrates that joint learning of document structure and answer extraction, combined with domain-specific pre-training, is highly effective for technical QA. While the model shows high versatility across devices, future work will explore generative QA to synthesize information from multiple sections and the integration of multi-modal data, such as diagrams and icons, which are prevalent in technical manuals.
Summary updated