BERT with History Answer Embedding for Conversational Question Answering

Chen Qu, Liu Yang, Minghui Qiu, W. Bruce Croft, Yongfeng Zhang, Mohit Iyyer

Abstract

Conversational search is an emerging topic in the information retrieval community. One of the major challenges to multi-turn conversational search is to model the conversation history to answer the current question. Existing methods either prepend history turns to the current question or use complicated attention mechanisms to model the history. We propose a conceptually simple yet highly effective approach referred to as history answer embedding. It enables seamless integration of conversation history into a conversational question answering (ConvQA) model built on BERT (Bidirectional Encoder Representations from Transformers). We first explain our view that ConvQA is a simplified but concrete setting of conversational search, and then we provide a general framework to solve ConvQA. We further demonstrate the effectiveness of our approach under this framework. Finally, we analyze the impact of different numbers of history turns under different settings to provide new insights into conversation history modeling in ConvQA.

Introduction

Standard Question Answering (QA) typically focuses on answering a single question based on a provided passage. However, in real-world scenarios, information seeking often occurs through multi-turn dialogues. Conversational Question Answering (ConvQA) addresses this by requiring models to understand the context established by previous turns. For instance, a follow-up question like "Why did people feel that way?" requires the system to identify the referent of "that way" from the preceding conversation history.

Current approaches to history modeling generally follow two strategies: prepending previous turns to the current query or employing complex recurrent attention mechanisms. Prepending is simple but inefficient, as it consumes BERT’s limited input sequence length (typically 512 tokens), often leading to the truncation of the source passage. Conversely, complex attention models like FlowQA are effective but computationally expensive and slow to train. This paper introduces History Answer Embedding (HAE), a method designed to be as simple as prepending while maintaining the robustness of complex attention models.

Methodology: History Answer Embedding (HAE)

The core innovation of this work is the HAE layer, which integrates conversation history directly into the BERT embedding stage. In a standard BERT model, a token's representation is the sum of its token, segment, and position embeddings. HAE introduces a fourth embedding layer specifically for tokens in the passage.

The HAE architecture integrates a history embedding layer into the standard BERT input representation.

For each token in the passage, the model learns two distinct embeddings: $E_H$, indicating the token was part of an answer in a previous turn, and $E_N$, indicating it was not. By "tagging" previous answers within the document text, HAE allows BERT’s self-attention mechanism to naturally prioritize historically relevant spans without increasing the input sequence length. This "length-efficient" approach preserves the maximum amount of space for the document itself.

Experimental Results

The model was evaluated on the QuAC (Question Answering in Context) dataset, which features information-seeking dialogues based on Wikipedia articles. Unlike factoid-based datasets, QuAC requires deeper contextual understanding across dialogues that can last up to 12 turns.

Performance and Efficiency

The BERT + HAE model achieved an F1 score of 62.4 on the QuAC leaderboard, significantly outperforming the BiDAF++ baseline (50.2) and a vanilla BERT model (54.4). Beyond accuracy, HAE demonstrated remarkable training efficiency. While complex models like FlowQA required nearly 57 hours to train, the HAE-augmented BERT model reached comparable performance in just 10 hours.

Robustness to History Length

A key finding of this research is the relationship between history length and model performance. Traditional prepending methods (e.g., PHQA - Prepend History Questions and Answers) tend to degrade in performance after just two turns of history. This degradation occurs because the input becomes increasingly noisy and the passage is truncated to fit length constraints.

Comparison of F1 scores across different numbers of history turns for HAE and prepending methods (PHQA, PHA).

As shown in the figure above, HAE remains stable and even improves as more history turns are included (up to 5-6 turns). This suggests that HAE effectively filters the "noise" of history by focusing on answer spans rather than raw text prepending.

Conclusion

History Answer Embedding provides a simple yet powerful way to adapt BERT for conversational tasks. By encoding history as a feature of the document tokens, HAE avoids the length-efficiency trade-offs of prepending and the computational overhead of recurrent architectures. Future work may involve developing learned history selection modules to further refine which past turns are most relevant to the current query.

Summary updated

Abstract

Conversational search is an emerging topic in the information retrieval community. One of the major challenges to multi-turn conversational search is to model the conversation history to answer the current question. Existing methods either prepend history turns to the current question or use complicated attention mechanisms to model the history. We propose a conceptually simple yet highly effective approach referred to as history answer embedding. It enables seamless integration of conversation history into a conversational question answering (ConvQA) model built on BERT (Bidirectional Encoder Representations from Transformers). We first explain our view that ConvQA is a simplified but concrete setting of conversational search, and then we provide a general framework to solve ConvQA. We further demonstrate the effectiveness of our approach under this framework. Finally, we analyze the impact of different numbers of history turns under different settings to provide new insights into conversation history modeling in ConvQA.
BibTeX
@inproceedings{qu2019bert,
  author = {Chen Qu and Liu Yang and Minghui Qiu and W. Bruce Croft and Yongfeng Zhang and Mohit Iyyer},
  title = {{BERT with History Answer Embedding for Conversational Question Answering}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2019},
  url = {https://openreview.net/forum?id=zHOhIeq6Sqj},
}
Figure