TL;DR
This paper introduces EMQAP, a novel multi-task learning framework for question answering (QA) over electronic device manuals (e-manuals). To address the lack of domain-specific datasets, they created a massive e-manual corpus for pre-training and several benchmark QA datasets, achieving significant improvements over existing methods.
What Problem Does This Paper Address?
Answering questions from instructional texts like e-manuals is a less explored area compared to open-domain QA. This is largely due to the absence of standardized datasets. E-manuals present unique challenges: instructions are often lengthy and sequential, answers can be spread across non-contiguous sections, and they require a deep understanding of domain-specific language. Existing QA models, trained on general text, struggle with these nuances.
Methods
The authors propose EMQAP, a framework built on two key components:
- Domain-Specific Pre-training: They first create a large corpus of 307,957 e-manuals (the "E-Manual Corpus") and use it to pre-train a RoBERTa model. This domain-adaptive pre-training helps the model understand the specific language and structure of e-manuals. They employ techniques like Elastic Weight Consolidation (EWC) and Learning Rate Decay (LRD) to retain general knowledge while adapting to the new domain.
- Multi-Task Learning (MTL) Framework: EMQAP uses a supervised MTL approach for the dual tasks of:
- Section Retrieval (SR): Identifying the most relevant section(s) within an e-manual that likely contain the answer.
- Answer Retrieval (AR): Extracting the precise answer span from the retrieved section. The SR and AR branches share the RoBERTa encoder, leveraging hard parameter sharing to reduce overfitting. The framework is designed to handle non-contiguous answers, a common characteristic of e-manuals.
Datasets
The paper introduces several new datasets:
- E-Manual Corpus: A large collection of 307,957 e-manuals used for pre-training.
- S10 QA Dataset: Manually curated question-answer pairs based on a Samsung S10 phone e-manual (904 pairs).
- Smart TV/Remote QA Dataset: Manually curated QA pairs from a Samsung Smart TV/Remote e-manual (950 pairs).
- Amazon Consumer Questions Forum Dataset: Questions extracted from Amazon forums related to e-manuals, filtered for answerability (1,028 answerable questions).
- Generic QA Dataset: 10 question-answer pairs for 40 different devices, sampled from the S10 QA dataset and Amazon forums, to test versatility.
They also compare their work against the TechQA Dataset, which is similar in domain but differs in question curation and answer length characteristics.
Key Insights and Ideas
- Domain-Specific Pre-training is Crucial: Pre-training on a large corpus of e-manuals significantly improves the model's understanding of domain-specific terminology and instructional language, leading to better performance.
- MTL for Section and Answer Retrieval: Combining SR and AR into a single multi-task learning framework allows the model to learn a more holistic understanding of the QA task, benefiting both components.
- Handling Non-Contiguous Answers: The architecture's ability to predict non-contiguous spans is vital for e-manuals, where answers might be scattered across sentences or paragraphs.
- Crowdsourcing for Quality Assurance: Using a crowdsourcing platform (Appen) with experienced workers helped validate the quality of expert annotations, ensuring the benchmark datasets are reliable.
Key Results
- EMQAP achieved significant improvements over competitive baselines, showing about a 40% increase in ROUGE-L F1 scores on the e-manual annotated QA pairs.
- The proposed TF-IDF + T5 method for unsupervised information retrieval (section selection) performed best among several baselines.
- In terms of exact match, EMQAP outperformed other baselines, with some models struggling to retrieve even a single correct answer.
- EMQAP demonstrated versatility across different device types and question categories (factual, procedural, location-based).
Limitations and Future Work
- Dataset Size for Pre-training: While large, the e-manual corpus is noted as "small" compared to general domain pre-training datasets, suggesting further scaling could be beneficial.
- Fine-tuning Specificity: EMQAP's performance on the Smart TV dataset was slightly inferior to the S10 dataset, indicating that fine-tuning on the target domain is important, though the drop was minimal, showing versatility.
- Future Work: The authors suggest that their extensive e-manual QA dataset is well-rounded and will be valuable for the research community. They also implicitly suggest further exploration of multi-task learning architectures and domain adaptation techniques.
Summary updated