Question Answering over Electronic Devices: A New Benchmark Dataset and a Multi-Task Learning based QA Framework

Abhilash Nandy, Soumya Sharma, Shubham Maddhashiya, Kapil Sachdeva, Pawan Goyal, Niloy Ganguly

TL;DR

This paper introduces EMQAP, a novel multi-task learning framework for question answering (QA) over electronic device manuals (e-manuals). To address the lack of domain-specific datasets, they created a massive e-manual corpus for pre-training and several benchmark QA datasets, achieving significant improvements over existing methods.

What Problem Does This Paper Address?

Answering questions from instructional texts like e-manuals is a less explored area compared to open-domain QA. This is largely due to the absence of standardized datasets. E-manuals present unique challenges: instructions are often lengthy and sequential, answers can be spread across non-contiguous sections, and they require a deep understanding of domain-specific language. Existing QA models, trained on general text, struggle with these nuances.

Methods

The authors propose EMQAP, a framework built on two key components:

  1. Domain-Specific Pre-training: They first create a large corpus of 307,957 e-manuals (the "E-Manual Corpus") and use it to pre-train a RoBERTa model. This domain-adaptive pre-training helps the model understand the specific language and structure of e-manuals. They employ techniques like Elastic Weight Consolidation (EWC) and Learning Rate Decay (LRD) to retain general knowledge while adapting to the new domain.
  2. Multi-Task Learning (MTL) Framework: EMQAP uses a supervised MTL approach for the dual tasks of:
  3. Section Retrieval (SR): Identifying the most relevant section(s) within an e-manual that likely contain the answer.
  4. Answer Retrieval (AR): Extracting the precise answer span from the retrieved section. The SR and AR branches share the RoBERTa encoder, leveraging hard parameter sharing to reduce overfitting. The framework is designed to handle non-contiguous answers, a common characteristic of e-manuals.

Datasets

The paper introduces several new datasets:

  • E-Manual Corpus: A large collection of 307,957 e-manuals used for pre-training.
  • S10 QA Dataset: Manually curated question-answer pairs based on a Samsung S10 phone e-manual (904 pairs).
  • Smart TV/Remote QA Dataset: Manually curated QA pairs from a Samsung Smart TV/Remote e-manual (950 pairs).
  • Amazon Consumer Questions Forum Dataset: Questions extracted from Amazon forums related to e-manuals, filtered for answerability (1,028 answerable questions).
  • Generic QA Dataset: 10 question-answer pairs for 40 different devices, sampled from the S10 QA dataset and Amazon forums, to test versatility.

They also compare their work against the TechQA Dataset, which is similar in domain but differs in question curation and answer length characteristics.

Key Insights and Ideas

  • Domain-Specific Pre-training is Crucial: Pre-training on a large corpus of e-manuals significantly improves the model's understanding of domain-specific terminology and instructional language, leading to better performance.
  • MTL for Section and Answer Retrieval: Combining SR and AR into a single multi-task learning framework allows the model to learn a more holistic understanding of the QA task, benefiting both components.
  • Handling Non-Contiguous Answers: The architecture's ability to predict non-contiguous spans is vital for e-manuals, where answers might be scattered across sentences or paragraphs.
  • Crowdsourcing for Quality Assurance: Using a crowdsourcing platform (Appen) with experienced workers helped validate the quality of expert annotations, ensuring the benchmark datasets are reliable.

Key Results

  • EMQAP achieved significant improvements over competitive baselines, showing about a 40% increase in ROUGE-L F1 scores on the e-manual annotated QA pairs.
  • The proposed TF-IDF + T5 method for unsupervised information retrieval (section selection) performed best among several baselines.
  • In terms of exact match, EMQAP outperformed other baselines, with some models struggling to retrieve even a single correct answer.
  • EMQAP demonstrated versatility across different device types and question categories (factual, procedural, location-based).

Limitations and Future Work

  • Dataset Size for Pre-training: While large, the e-manual corpus is noted as "small" compared to general domain pre-training datasets, suggesting further scaling could be beneficial.
  • Fine-tuning Specificity: EMQAP's performance on the Smart TV dataset was slightly inferior to the S10 dataset, indicating that fine-tuning on the target domain is important, though the drop was minimal, showing versatility.
  • Future Work: The authors suggest that their extensive e-manual QA dataset is well-rounded and will be valuable for the research community. They also implicitly suggest further exploration of multi-task learning architectures and domain adaptation techniques.

Summary updated

Abstract

Answering questions asked from instructional corpora such as E-manuals, recipe books, etc., has been far less studied than open-domain factoid context-based question answering. This can be primarily attributed to the absence of standard benchmark datasets. In this paper, we meticulously create a large amount of data connected with E-manuals and develop a suitable algorithm to exploit it. We collect E-Manual Corpus, a huge corpus of 307,957 E-manuals, and pretrain RoBERTa on this large corpus. We create various benchmark QA datasets which include question answer pairs curated by experts based upon two E-manuals, real user questions from Community Question Answering Forum pertaining to E-manuals etc. We introduce EMQAP (E-Manual Question Answering Pipeline) that answers questions pertaining to electronics devices. Built upon the pretrained RoBERTa, it harbors a supervised multi-task learning framework which efficiently performs the dual tasks of identifying the section in the E-manual where the answer can be found and the exact answer span within that section. For E-Manual annotated question-answer pairs, we show an improvement of about 40% in ROUGE-L F1 scores over most competitive baseline. We perform a detailed ablation study and establish the versatility of EMQAP across different circumstances. The code and datasets are shared at https://github.com/abhi1nandy2/EMNLP-2021-Findings, and the corresponding project website is https://sites.google.com/view/emanualqa/home.
BibTeX
@inproceedings{nandy2021question,
  author = {Abhilash Nandy and Soumya Sharma and Shubham Maddhashiya and Kapil Sachdeva and Pawan Goyal and Niloy Ganguly},
  title = {{Question Answering over Electronic Devices: A New Benchmark Dataset and a Multi-Task Learning based QA Framework}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.392},
}
Figure