Generating Extractive Answers: Gated Recurrent Memory Reader for Conversational Question Answering

Xuanyu Zhang, Qing YangEMNLP 2023 · OpenReview

Introduction

Conversational Question Answering (CQA) presents a unique challenge compared to traditional single-turn machine reading comprehension (MRC). In CQA, questions are often interdependent; a model must understand the context of previous turns to resolve coreferences and ellipses. For instance, answering "Is she carrying something?" requires knowing who "she" refers to from earlier in the dialogue.

Most existing CQA models address this by concatenating previous questions and answers into the current input. However, this approach is often superficial and redundant, leading to a massive memory footprint and potential confusion for the model as the conversation grows. This paper introduces the Gated Recurrent Memory Reader (GRMR), a novel architecture designed to model historical memory deeply and selectively while maintaining a lean storage profile.

Methodology: The Gated Recurrent Memory Reader

The GRMR framework reimagines CQA as a recurrent sequence-to-sequence task. Instead of simply matching text, it "generates" extractive answers by predicting start and end pointers within the passage turn by turn.

The GRMR architecture, featuring a Passage Encoder and a Question Decoder with Gated Memory.

The architecture consists of two primary components:

  1. Passage Encoder: Encodes the input text into an intermediate representation.
  2. Question Decoder with Gated Memory: This is the core innovation. Unlike models that re-encode history every turn, GRMR maintains a hidden state that evolves. It uses a Gated Memory Layer (similar to GRU/LSTM mechanisms) to decide which parts of the passage and previous questions remain relevant. This allows the model to "forget" distractions and focus on the current conversational thread.

The model also employs an interaction layer with attention mechanisms to align the current question with the passage, followed by a pointer network decoder to identify the answer span.

Experimental Results

The researchers evaluated GRMR on the CoQA (Conversational Question Answering) dataset, which includes 127k questions across diverse domains.

Performance and Efficiency

GRMR achieved an F1 score of 66.3 on the CoQA development set, outperforming other non-pretrained baselines such as DrQA+PGNet (66.2). While it does not yet match the performance of massive pre-trained models like BERT, it demonstrates the power of efficient memory management.

The most significant advantage of GRMR is its storage efficiency. By avoiding the redundant concatenation of history, GRMR maintains a constant, lean profile.

Storage space comparison between GRMR and traditional concatenation methods across different context lengths.

As shown in the data, traditional models require up to 4x more storage space than GRMR when handling a 3-turn context. This makes GRMR particularly suitable for deployment in resource-constrained environments.

Ablation Study

Ablation experiments confirmed the importance of the gated memory mechanism. Removing the gated memory caused the F1 score to drop significantly from 66.3 to 58.8, proving that the recurrent memory—not just the base encoder—is doing the heavy lifting for conversational understanding.

Conclusion and Future Work

The Gated Recurrent Memory Reader offers a smarter, more efficient way to handle the complexities of conversational QA. By treating history as a selective, evolving memory rather than a growing text block, it achieves competitive performance with a fraction of the storage requirements. Future work could involve integrating pre-trained Transformer encoders to close the performance gap with state-of-the-art models and extending the framework to handle generative (non-extractive) answers.

Summary updated

Abstract

Conversational question answering (CQA) is a more complicated task than traditional single-turn machine reading comprehension (MRC). Different from large language models (LLMs) like ChatGPT, the models of CQA need to extract answers from given contents to answer follow-up questions according to conversation history. In this paper, we propose a novel architecture, i.e., Gated Recurrent Memory Reader (GRMR), which integrates traditional extractive MRC models into a generalized sequence-to-sequence framework. After the passage is encoded, the decoder will generate the extractive answers turn by turn. Different from previous models that concatenate the previous questions and answers as context superficially and redundantly, our model can use less storage space and consider historical memory deeply and selectively. Experiments on the Conversational Question Answering (CoQA) dataset show that our model achieves comparable results to most models with the least space occupancy.
BibTeX
@inproceedings{zhang2023generating,
  author = {Xuanyu Zhang and Qing Yang},
  title = {{Generating Extractive Answers: Gated Recurrent Memory Reader for Conversational Question Answering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=WxxYSpsv97},
}
Figure