TL;DR
This paper tackles the challenge of generating answers for reading comprehension tasks where the answer isn't a single, contiguous span within the text. They propose a "multi-span" approach that intelligently combines multiple extracted text segments to form a more accurate and natural-sounding answer, outperforming traditional extractive and generative methods, especially for longer answers.
What Problem Does This Paper Address?
Machine Reading Comprehension (MRC) aims to answer questions based on provided text. Traditionally, MRC tasks are divided into two categories: extractive (where the answer is a direct span from the text) and non-extraction (where the answer might be synthesized or paraphrased). While extractive methods using pointer networks excel at finding precise spans, they struggle when answers are complex or require combining information. Generative methods, on the other hand, can produce novel answers but often suffer from grammatical errors or semantic incoherence, especially for longer responses. This paper addresses the limitations of both by proposing a method to construct answers from multiple, potentially discontinuous, spans within the source text, aiming for the best of both worlds.
Methods
The core of the proposed method is a "multi-span decoder." Unlike traditional pointer networks that predict just the start and end of a single answer span, this model first identifies multiple high-confidence "candidate spans" within the passage. These candidate spans are learned by assigning a score to each word in the passage, indicating its likelihood of being part of an answer.
The method then employs two strategies for selecting these candidate spans:
- N-gram Spans: This approach considers all contiguous sequences of words (n-grams) up to a maximum length. A penalty is applied to longer n-grams to favor more concise phrases.
- Syntactic Spans: This method leverages syntactic parse trees of the passage to identify meaningful phrases or constituents as candidate spans. This aims to ensure that the selected spans are linguistically coherent.
Once candidate spans are identified and scored, they are combined in their original order within the passage to form the final answer. This "composing" approach allows for answers that are not necessarily contiguous.
Datasets
The primary dataset used for training and evaluation in this paper is MS MARCO. This dataset is noted for its non-extraction nature, where questions are derived from search engine queries, and answers are human-generated, often requiring synthesis from multiple passages. The authors also created an "answerable subset" of MS MARCO for their main evaluation, excluding non-answerable questions to focus on the answer generation aspect. They also split the official development set into their own development and test sets.
Key Insights and Ideas
- The "sweet spot" for answer composition: The paper highlights that answers are often composed of multiple, short, meaningful phrases rather than a single long, extracted segment.
- Syntactic structure matters: Using syntactic parse trees to identify candidate spans leads to better performance than relying solely on n-grams, suggesting that linguistic structure is crucial for generating coherent answers.
- Longer answers benefit most: The multi-span approach shows a clear advantage in generating longer answers, as these are more likely to require combining information from different parts of the text.
- Balancing precision and recall: The choice of parameters for span selection (e.g., threshold values) involves a trade-off between metrics like BLEU (precision-focused) and ROUGE-L (recall-focused), indicating the complexity of optimizing for natural language generation.
Key Results
The proposed multi-span decoder significantly outperforms both one-span extractive decoders and standard Seq2Seq generative decoders on the MS MARCO dataset.
- Overall Performance: The multi-span approach achieved higher scores across all evaluated metrics (BLEU-1 to BLEU-4, and ROUGE-L) compared to baselines.
- N-gram vs. Syntactic Spans: Syntactic span selection generally yielded better results than n-gram span selection, demonstrating the value of linguistic structure.
- Answer Length: The model's performance improved with longer answers, with scores for answers between 21-25 words and over 25 words being particularly high.
- Decoding Speed: While the syntactic span decoder requires pre-processing (parsing), the n-gram span decoder is significantly faster, running thousands of times faster than the syntactic version.
Limitations and Future Work
- Syntactic Parsing Overhead: The syntactic span selection method is computationally more expensive due to the need for pre-trained parsers, which can slow down the decoding process. The authors note that their adopted parser is particularly inefficient.
- Handling Short Answers: The model performs poorly on very short answers (0-5 words), especially those that are simple affirmations like "YES" or "NO," which are less likely to be composed from multiple spans.
- Future Work: The paper suggests that further improvements could come from more optimized passage ranking modules and encoders, especially for tasks involving multiple passages. The authors also implicitly point towards exploring more efficient parsing methods or alternative ways to capture linguistic structure for faster and more robust multi-span selection.
Summary updated