Answer Generation through Unified Memories over Multiple Passages

Makoto Nakatsuji, Sohei OkuiIJCAI 2020 · DOI

TL;DR

This paper introduces GUM-MP, a novel neural network for generating answers from multiple text passages. GUM-MP addresses the challenge of answer generation by explicitly modeling relationships between passages and using "positive" and "negative" passages to better identify crucial information, leading to more accurate answers.

What Problem Does This Paper Address?

Modern AI systems are increasingly capable of "reading" and answering questions based on provided text. However, when faced with multiple text passages that might contain overlapping or even conflicting information, existing methods struggle. They often get confused by irrelevant details or subtly different descriptions, leading to inaccurate answers. This paper tackles the problem of effectively synthesizing information from multiple passages to generate a coherent and accurate answer, going beyond simply extracting text spans.

Methods

The core innovation of GUM-MP lies in its "Unified Memories" (UMs) and its use of positive and negative passages.

  • Positive and Negative Passage Analysis: The model distinguishes between passages that are relevant to the question ("positive") and those that are not ("negative"). By comparing how tokens in positive passages match the question versus how tokens in negative passages match, GUM-MP can filter out irrelevant information and focus on truly important tokens.
  • Multi-Perspective Memories (MPMs): For each passage, the model encodes token embeddings by matching them against the question from multiple perspectives. This captures richer semantic relationships.
  • Passage Alignment Memory (PAM): GUM-MP explicitly models the relationships between passages. It aligns hidden vectors from other passages into a latent semantic space (PAM) to capture correlated topics.
  • Unified Memories (UMs): The final memory representation for each passage (UM) combines the passage's MPM with information about its alignment to other passages. This allows the model to understand how different pieces of information across passages relate to each other.
  • Encoder-Decoder with Multiple-Pointer-Generator: The model uses a standard encoder-decoder architecture but extends the pointer-generator mechanism to handle multiple passages. This allows the decoder to point to important tokens within the passages, informed by the UMs.

Datasets

The paper evaluates GUM-MP on two datasets:

  • MS-MARCO: A large-scale dataset of user queries from the Bing search engine, with approximately ten passages and one answer per question. The authors used the natural language generation subset.
  • Oshiete-goo: A Japanese community QA dataset focusing on "love advice." This dataset is noted for having diverse QA pairs and potentially more overlapping topics.

Key Insights and Ideas

  • The Importance of Negative Passages: Explicitly using irrelevant passages as a "negative" signal is crucial for filtering noise and identifying truly important information. This is a key differentiator from prior work.
  • Inter-Passage Relationships Matter: Simply processing passages independently and then merging results is insufficient. GUM-MP's approach of aligning passage memories (PAM) and creating unified memories (UMs) effectively captures how topics are discussed across different texts.
  • "Unified Memories" as a Semantic Bridge: The UMs act as a powerful representation that consolidates information about a passage's content, its relevance to the question, and its relationship with other passages, leading to better answer generation.
  • Beyond Extraction: The paper emphasizes generating answers, not just extracting them. The pointer-generator mechanism, informed by UMs, allows for more flexible and accurate answer construction.

Key Results

GUM-MP consistently outperformed existing state-of-the-art methods on both MS-MARCO and Oshiete-goo datasets across BLEU-1 and ROUGE-L metrics. For instance, on MS-MARCO, GUM-MP achieved a BLEU-1 score of 0.503, significantly higher than MPQG (0.364) and S-Net (0.407). Ablation studies showed that both the "negative passage" analysis and the "Unified Memories" components were essential for GUM-MP's superior performance.

Limitations and Future Work

  • Computational Complexity: While not explicitly detailed as a limitation, the multi-perspective matching and passage alignment likely introduce significant computational overhead.
  • Future Work: The authors plan to apply their ideas to a Transformer-based encoder-decoder model, suggesting a potential for further improvements by leveraging more advanced architectures.

Summary updated

BibTeX
@inproceedings{nakatsuji2020answer,
  author = {Makoto Nakatsuji and Sohei Okui},
  title = {{Answer Generation through Unified Memories over Multiple Passages}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/529},
}
Figure