This paper, presented at SIGIR 2019, introduces a simple yet powerful modification to the BERT architecture to handle the complexities of multi-turn conversational search. By "highlighting" previous answers directly within the document text using a new embedding layer, the authors achieve state-of-the-art performance with significantly higher efficiency than previous complex attention-based models.
TL;DR
The authors propose History Answer Embedding (HAE), a method that integrates conversation history into BERT by adding a simple embedding layer that marks which parts of a document were answers to previous questions. This approach outperforms traditional "history prepending" methods and is five times faster to train than complex recurrent models like FlowQA.
What Problem Does This Paper Address?
Standard Question Answering (QA) usually involves a single question about a single passage. Conversational Question Answering (ConvQA) is harder because questions are part of a dialogue. A user might ask, "Was he intelligent?" followed by "Why did people feel that way?" To answer the second question, the system must know who "he" is and what "that way" refers to by looking at the conversation history.
Existing solutions generally fall into two camps:
- Prepending: Sticking previous questions and answers onto the front of the current question. This is simple but quickly hits BERT’s maximum sequence length limit (usually 512 tokens), leaving less room for the actual document.
- Complex Attention: Using sophisticated recurrent structures (like FlowQA) to pass "latent" information between turns. These are effective but computationally expensive and slow to train.
The authors sought a "middle way": a method as simple as prepending but as robust as complex attention.
Methods
The core innovation is the History Answer Embedding (HAE) layer.
In a standard BERT model, a token’s representation is the sum of its Token, Segment, and Position embeddings. The authors add a fourth: the HAE layer.
- For every token in the passage, the model learns two unique embeddings: $E_H$ (the token was part of a previous answer) and $E_N$ (the token was not).
- This effectively "tags" the document, telling BERT: "Pay attention to these specific spans; they were relevant a moment ago."
Because this information is added at the embedding level, it doesn't increase the input sequence length. It allows BERT’s self-attention mechanism to naturally incorporate history when processing the current question. For history selection, the authors used a simple rule-based approach, typically looking back at the 5 or 6 most recent turns.
Datasets
The researchers evaluated their model on QuAC (Question Answering in Context).
- QuAC vs. others: Unlike the CoQA dataset, which is often factoid-based, QuAC contains non-factoid, information-seeking dialogues where the "seeker" only sees the title of a Wikipedia page and must explore the content through dialogue.
- Scale: QuAC contains roughly 11K dialogues and 83K questions, with dialogues lasting up to 12 turns.
Key Insights and Ideas
- The "Length-Efficiency" Trade-off: BERT has a hard limit on input length. Prepending history is "length-inefficient" because it consumes the budget that should be used for the document. HAE is "length-efficient" because it encodes history as a feature of the existing tokens.
- Answers Matter More Than Questions: The authors found that prepending just previous answers (PHA) was nearly as effective as prepending both questions and answers (PHQA). This suggests that the "grounding" provided by previous answers is the most critical signal for the model.
- Robustness to Long History: Prepending methods actually get worse as you add more history (after about 2 turns) because the input becomes too noisy and the document gets truncated. HAE, however, stays stable or improves even when looking back 6+ turns.
Key Results
- Performance: The BERT + HAE model achieved an F1 score of 62.4 on the QuAC leaderboard, a significant jump over the BiDAF++ baseline (50.2) and the vanilla BERT model (54.4).
- Efficiency: HAE is remarkably fast. It trained in about 10 hours, compared to 56.8 hours for FlowQA, while achieving comparable or better results.
- Statistical Significance: The authors used a Student’s paired t-test to confirm that HAE’s improvements over prepending methods were statistically significant ($p < 0.05$).
Limitations and Future Work
- Rule-Based Selection: The current model uses a simple "last $j$ turns" rule for history. The authors acknowledge that a learned "history selection" module—one that identifies which specific past turns are actually relevant to the current query—could further improve performance.
- BERT Constraints: While HAE saves space, the model is still bound by BERT's overall architecture and sequence limits.
- Integration: Future work could involve integrating HAE with more advanced history selection strategies to handle even longer, more complex information-seeking sessions.
Summary updated