The Challenge of Conversational QA
Building a conversational agent that can answer questions using a Knowledge Base (KB) is notoriously difficult. Unlike standard Question Answering (QA), conversational QA involves multi-turn interactions where users rarely provide full context in every sentence.
Two primary hurdles make this challenging:
- Linguistic Context: Conversations are full of ellipsis (omitting words because they are implied, e.g., "How about Harvard?") and co-reference (using pronouns like "he" or "it" to refer to previous entities). A model must "remember" the history to understand the current intent.
- Weak Supervision at Scale: In realistic settings, we only have the final answer (the "denotation"), not the specific logical query (the "logical form") used to get it. Training a model to find the right query in a KB haystack of 12.8 million entities without explicit query labels is a massive search problem.
To address these issues, the "Dialog-to-Action" (D2A) framework maps conversational questions into executable logical forms. By incorporating a novel "dialog memory management" component, the model successfully resolves complex linguistic phenomena, significantly outperforming previous memory-network baselines.
The Dialog-to-Action (D2A) Framework
The authors propose a semantic parsing-based approach consisting of three main pillars:
1. Grammar-Guided Decoder
Instead of predicting raw text, the decoder predicts a sequence of "actions" (e.g., find, count, union) based on a predefined, flexible grammar. This ensures that the generated output is always a valid, executable query. By restricting the decoder to a specific grammar, the authors effectively prune the search space, making it feasible to operate over a massive KB.
2. Dialog Memory Management
This is the core innovation of the paper. The model maintains a memory of historical context to resolve missing information in current turns.
The memory tracks:
- Entities: Extracted from previous questions and answers.
- Predicates: Relations mentioned in the previous turn.
- Action Subsequences: Partial logical forms from previous turns.
When a user asks an elliptical question, they are often asking for the same logic applied to a new entity. D2A can copy the structure of a previous query (Action Subsequence Memory) and simply swap out the subject by instantiating historical elements.
3. Learning from Denotations
To train without "gold" logical forms, the authors use a breadth-first search (BFS) algorithm to find action sequences that lead to the correct answer in the KB. The model is then trained to maximize the likelihood of these successful sequences.
Datasets and Key Results
The model was evaluated on the CSQA (Complex Sequential Question Answering) dataset, which contains 200K dialogs (1.6M turns) over a Wikidata-based KB featuring 12.8M entities.
The D2A model significantly outperformed the previous state-of-the-art baseline, HRED+KVmem (a hierarchical encoder-decoder with key-value memory):
- Massive Performance Gains: D2A achieved an overall Recall of 64.04% and Precision of 61.76%, compared to the baseline's 18.40% Recall and 6.30% Precision.
- Ellipsis Handling: In the "Simple Question (Ellipsis)" category, D2A reached 86.58% Recall. Ablation studies showed that removing the "Predicate Memory" caused this to plummet to ~15%, proving that remembering previous relations is vital for resolving omitted context.
- Co-reference Resolution: For questions involving pronouns, the "Entity Memory" was crucial; removing it dropped accuracy from ~71% to ~38%.
These results highlight a key insight: semantic parsing (converting text to logic) is inherently more powerful for complex reasoning than text-to-text generation. By explicitly manipulating functions and entities, the model can perform set operations that are difficult for standard Seq2Seq models.
Limitations and Future Work
While highly effective, the authors highlight several areas for improvement:
- Entity Linking: The model struggles when different entities share the same name (surface form). Better word representations incorporating KB context could help.
- Spurious Programs: The BFS training method sometimes finds "lucky" queries—logical forms that get the right answer for the wrong reason.
- Error Propagation: Because the model replicates previous action sequences, an error in an early turn can cascade through the entire conversation.
- Grammar Coverage: Expanding the action set to cover more complex queries (e.g., specific diplomatic relations) without exploding the search space remains a future goal.
Summary updated