Composing Answer from Multi-spans for Reading Comprehension

Zhuosheng Zhang, Yiqing Zhang, Hai Zhao, Xi Zhou

TL;DR

This paper tackles the challenge of generating answers for reading comprehension tasks where the answer isn't a single, contiguous span within the text. They propose a "multi-span" approach that intelligently combines multiple extracted text segments to form a more accurate and natural-sounding answer, outperforming traditional extractive and generative methods, especially for longer answers.

What Problem Does This Paper Address?

Machine Reading Comprehension (MRC) aims to answer questions based on provided text. Traditionally, MRC tasks are divided into two categories: extractive (where the answer is a direct span from the text) and non-extraction (where the answer might be synthesized or paraphrased). While extractive methods using pointer networks excel at finding precise spans, they struggle when answers are complex or require combining information. Generative methods, on the other hand, can produce novel answers but often suffer from grammatical errors or semantic incoherence, especially for longer responses. This paper addresses the limitations of both by proposing a method to construct answers from multiple, potentially discontinuous, spans within the source text, aiming for the best of both worlds.

Methods

The core of the proposed method is a "multi-span decoder." Unlike traditional pointer networks that predict just the start and end of a single answer span, this model first identifies multiple high-confidence "candidate spans" within the passage. These candidate spans are learned by assigning a score to each word in the passage, indicating its likelihood of being part of an answer.

The method then employs two strategies for selecting these candidate spans:

  1. N-gram Spans: This approach considers all contiguous sequences of words (n-grams) up to a maximum length. A penalty is applied to longer n-grams to favor more concise phrases.
  2. Syntactic Spans: This method leverages syntactic parse trees of the passage to identify meaningful phrases or constituents as candidate spans. This aims to ensure that the selected spans are linguistically coherent.

Once candidate spans are identified and scored, they are combined in their original order within the passage to form the final answer. This "composing" approach allows for answers that are not necessarily contiguous.

Datasets

The primary dataset used for training and evaluation in this paper is MS MARCO. This dataset is noted for its non-extraction nature, where questions are derived from search engine queries, and answers are human-generated, often requiring synthesis from multiple passages. The authors also created an "answerable subset" of MS MARCO for their main evaluation, excluding non-answerable questions to focus on the answer generation aspect. They also split the official development set into their own development and test sets.

Key Insights and Ideas

  • The "sweet spot" for answer composition: The paper highlights that answers are often composed of multiple, short, meaningful phrases rather than a single long, extracted segment.
  • Syntactic structure matters: Using syntactic parse trees to identify candidate spans leads to better performance than relying solely on n-grams, suggesting that linguistic structure is crucial for generating coherent answers.
  • Longer answers benefit most: The multi-span approach shows a clear advantage in generating longer answers, as these are more likely to require combining information from different parts of the text.
  • Balancing precision and recall: The choice of parameters for span selection (e.g., threshold values) involves a trade-off between metrics like BLEU (precision-focused) and ROUGE-L (recall-focused), indicating the complexity of optimizing for natural language generation.

Key Results

The proposed multi-span decoder significantly outperforms both one-span extractive decoders and standard Seq2Seq generative decoders on the MS MARCO dataset.

  • Overall Performance: The multi-span approach achieved higher scores across all evaluated metrics (BLEU-1 to BLEU-4, and ROUGE-L) compared to baselines.
  • N-gram vs. Syntactic Spans: Syntactic span selection generally yielded better results than n-gram span selection, demonstrating the value of linguistic structure.
  • Answer Length: The model's performance improved with longer answers, with scores for answers between 21-25 words and over 25 words being particularly high.
  • Decoding Speed: While the syntactic span decoder requires pre-processing (parsing), the n-gram span decoder is significantly faster, running thousands of times faster than the syntactic version.

Limitations and Future Work

  • Syntactic Parsing Overhead: The syntactic span selection method is computationally more expensive due to the need for pre-trained parsers, which can slow down the decoding process. The authors note that their adopted parser is particularly inefficient.
  • Handling Short Answers: The model performs poorly on very short answers (0-5 words), especially those that are simple affirmations like "YES" or "NO," which are less likely to be composed from multiple spans.
  • Future Work: The paper suggests that further improvements could come from more optimized passage ranking modules and encoders, especially for tasks involving multiple passages. The authors also implicitly point towards exploring more efficient parsing methods or alternative ways to capture linguistic structure for faster and more robust multi-span selection.

Summary updated

Abstract

This paper presents a novel method to generate answers for non-extraction machine reading comprehension (MRC) tasks whose answers cannot be simply extracted as one span from the given passages. Using a pointer network-style extractive decoder for such type of MRC may result in unsatisfactory performance when the ground-truth answers are given by human annotators or highly re-paraphrased from parts of the passages. On the other hand, using generative decoder cannot well guarantee the resulted answers with well-formed syntax and semantics when encountering long sentences. Therefore, to alleviate the obvious drawbacks of both sides, we propose an answer making-up method from extracted multi-spans that are learned by our model as highly confident $n$-gram candidates in the given passage. That is, the returned answers are composed of discontinuous multi-spans but not just one consecutive span in the given passages anymore. The proposed method is simple but effective: empirical experiments on MS MARCO show that the proposed method has a better performance on accurately generating long answers, and substantially outperforms two competitive typical one-span and Seq2Seq baseline decoders.
BibTeX
@misc{zhang2020composing,
  author = {Zhuosheng Zhang and Yiqing Zhang and Hai Zhao and Xi Zhou and Xiang Zhou},
  title = {{Composing Answer from Multi-spans for Reading Comprehension}},
  year = {2020},
  eprint = {2009.06141},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.06141},
}
Figure