Answering Naturally : Factoid to Full length Answer Generation

Vaishali Pal, Manish Shrivastava, Irshad Ahmad Bhat2019 · DOI

Abstract

Modern reading comprehension systems excel at extracting short spans of text as answers, but these "factoids" often provide a robotic user experience in conversational interfaces. We propose a "plug-and-play" full-length answer generator that transforms these short spans into natural sentences using only the question and the extracted factoid. Utilizing a dual-encoder pointer-generator network and a new dataset of 315,000 triples, our system achieves a 74.05 BLEU score and an 86.25 ROUGE-L score, demonstrating significant improvements in conversational fluency without requiring access to the original source passage.

Introduction: Beyond Robotic Responses

Extractive Question Answering (QA) systems, typically trained on datasets like SQuAD, are highly effective at identifying specific entities or phrases within a document. However, responding to a user with an isolated phrase—such as answering "10th and 11th centuries" to a question about the Normans—feels abrupt. While template-based systems can mitigate this in narrow domains, they lack the flexibility required for general-purpose dialogue.

The goal of this work is to generate grammatically correct, fluent sentences from [Question + Factoid] pairs. This approach allows any extractive QA system to become more "natural" by simply appending this generation module.

Examples of the transformation from question and factoid to a natural target sentence.

Methodology: Dual-Encoder Pointer-Generator

The core challenge in this task is maintaining factual accuracy while ensuring linguistic variety. To address this, we employ a Pointer-Generator Network with a unique dual-encoder architecture.

Architecture Overview

Standard sequence-to-sequence models often struggle with Out-of-Vocabulary (OOV) words, such as rare names or technical terms. Our model overcomes this by using:

  1. Dual Encoders: Two separate 3-layered bidirectional LSTMs process the question and the factoid answer independently. This separation is vital because the syntactic relationship between a question and its answer is often indirect.
  2. Pointer Mechanism: The model calculates a generation probability ($P_{gen}$). It can either generate a common word from its fixed vocabulary or "point" to (copy) a word directly from the input sequence. This ensures that specific factual details like dates and names are preserved exactly.
The 2-Encoder Pointer-Generator architecture, showing independent encoders for the question and factoid.

Dataset: Scaling Natural Language Generation

To train this model, we developed a large-scale dataset of 315,000 [Question, Factoid, Full Answer] triples:

  • Automatic Subset (~300,000): Generated by identifying sentences in source passages that contained the factoid and shared high overlap with the question.
  • Manual Subset (15,000): Human-annotated samples providing complex sentence structures, including active and passive voice variations.

This hybrid dataset allows the model to learn the broad patterns of sentence restructuring from the large noisy set while refining its grammatical nuances from the high-quality manual set.

Key Results and Insights

Our evaluation shows that the system generalizes well across different domains, including knowledge bases (Freebase) and news articles (NewsQA).

  • Metrics: The augmented model achieved a 74.05 BLEU and 86.25 ROUGE-L score on the manual test set.
  • Context-Free Success: A significant finding is that the original source passage is not required for high-quality generation; the question itself contains sufficient structural information to rebuild a declarative sentence.
  • Robustness: The neural approach significantly outperformed traditional rule-based templates, especially when dealing with multi-clause questions or varied syntax.

Conclusion and Future Directions

By decoupling answer generation from the source passage, we provide a flexible tool for enhancing the conversational quality of QA systems. While the model occasionally faces challenges with tense agreement and coreference resolution, future iterations utilizing Transformer architectures and explicit coreference modules promise even greater fluency and accuracy.

Summary updated

Abstract

In recent years, the task of Question Answering over passages, also pitched as a reading comprehension, has evolved into a very active research area. A reading comprehension system extracts a span of text, comprising of named entities, dates, small phrases, etc., which serve as the answer to a given question. However, these spans of text would result in an unnatural reading experience in a conversational system. Usually, dialogue systems solve this issue by using template-based language generation. These systems, though adequate for a domain specific task, are too restrictive and predefined for a domain independent system. In order to present the user with a more conversational experience, we propose a pointer generator based full-length answer generator which can be used with most QA systems. Our system generates a full-length answer given a question and the extracted factoid/span answer without relying on the passage from where the answer was extracted. We also present a dataset of 315,000 question, factoid answer and full-length answer triples. We have evaluated our system using ROUGE-1,2,L and BLEU and achieved 74.05 BLEU score and 86.25 Rogue-L score.
BibTeX
@misc{pal2019answering,
  author = {Vaishali Pal and Manish Shrivastava and Irshad Ahmad Bhat},
  title = {{Answering Naturally : Factoid to Full length Answer Generation}},
  year = {2019},
  doi = {10.18653/v1/d19-5401},
}
Figure