Abstract
Question answering (QA) based on machine reading comprehension has seen a recent surge in popularity, yet most work has focused on extractive methods. We instead address a more challenging QA problem of generating a well-formed answer by reading and summarizing the paragraph for a given question. For the generative QA task, we introduce a new neural architecture, LatentQA, in which a novel stochastic selector network composes a well-formed answer with words selected from the question, the paragraph, and the global vocabulary, based on a sequence of discrete latent variables. Bayesian inference for the latent variables is performed to train the LatentQA model. The experiments on public datasets of natural answer generation confirm the effectiveness of LatentQA in generating high-quality well-formed answers.
The Challenge: Beyond Text Extraction
Most current Machine Reading Comprehension (MRC) systems are extractive: they identify a specific "span" (a start and end point) within a text to serve as an answer. While efficient, this approach fails for "How" or "Why" questions that require synthesis or summarization. Furthermore, extracted snippets often lack context, making them awkward for digital assistants to read aloud.
The challenge in generative QA is twofold:
- Source Ambiguity: An answer word might need to come from the paragraph (facts), the question (contextual framing), or a general vocabulary (connective tissue).
- Training Labels: Most generative models require expensive, manually labeled "answer spans" to learn where to look.
LatentQA addresses these by treating the source of each word as a latent variable, allowing the model to learn the best source for each word automatically during training.
Methodology: The Stochastic Selector Network
LatentQA is built on an augmented sequence-to-sequence (seq2seq) architecture. Its core innovation is the Stochastic Selector Network, which manages the synthesis of the final response.
The Three-Way Switch
At every step of generating an answer, the model uses a discrete latent variable ($y_t$) to decide between three sources:
- Question Source: Copying words from the user's query to maintain context.
- Paragraph Source: Copying factual words from the reference text.
- Global Vocabulary: Generating common connective words (e.g., "is," "the," "because").
Modeling Uncertainty and Bayesian Inference
Unlike deterministic models, LatentQA uses a continuous latent representation ($h_t$) to model the inherent ambiguity in natural language. This helps the model handle cases where a question might have multiple valid interpretations. To train the model, the authors use Variational Inference, employing the reparameterization trick for continuous variables and the Gumbel-Softmax trick to make the discrete "source selection" differentiable, allowing end-to-end training via backpropagation.
Experimental Evaluation
The model was evaluated on two major benchmarks: MS MARCO (V2.1), based on Bing search queries, and DuReader, a large-scale Chinese document-based QA dataset.
Key Results
LatentQA significantly outperformed existing state-of-the-art models (like S-Net and VNET) across both automated metrics and human evaluation:
- Automated Metrics: On MS MARCO, it achieved a Rouge-L of 50.97 and Bleu-1 of 45.48, outperforming the previous best generative model (S-Net+Seq2Seq) by over 4 points in Rouge-L.
- Human Evaluation: In blind tests, LatentQA was rated significantly higher for "Well-formedness" (4.61/5.0) and "Correctness" (4.09/5.0) compared to extractive models like BiDAF, which struggled with a well-formedness score of only 2.66.
Insights
- The Question as a Resource: Including the question as a copy source significantly boosts "well-formedness" by allowing the answer to mirror the query's structure.
- Stochasticity as a Regularizer: By marginalizing over latent variables, the model becomes more robust against overfitting, providing a natural safeguard when training on smaller datasets.
Conclusion and Future Work
LatentQA demonstrates that treating word sources as latent variables allows for the discovery of complex answer structures without explicit supervision. While the model excels at creating natural-sounding responses, future work may involve integrating external knowledge bases to answer questions requiring facts not present in the immediate text, and addressing the slight "syntactic correctness" gap where purely extractive models occasionally benefit from copying human-written snippets perfectly.
Summary updated