Dialog-to-Action: Conversational Question Answering Over a Large-Scale Knowledge Base

Daya Guo, Duyu Tang, Nan Duan, Ming Zhou, Jian YinNeurIPS 2018 · OpenReview

The Challenge of Conversational QA

Building a conversational agent that can answer questions using a Knowledge Base (KB) is notoriously difficult. Unlike standard Question Answering (QA), conversational QA involves multi-turn interactions where users rarely provide full context in every sentence.

Example of conversational QA with ellipsis and co-reference

Two primary hurdles make this challenging:

  1. Linguistic Context: Conversations are full of ellipsis (omitting words because they are implied, e.g., "How about Harvard?") and co-reference (using pronouns like "he" or "it" to refer to previous entities). A model must "remember" the history to understand the current intent.
  2. Weak Supervision at Scale: In realistic settings, we only have the final answer (the "denotation"), not the specific logical query (the "logical form") used to get it. Training a model to find the right query in a KB haystack of 12.8 million entities without explicit query labels is a massive search problem.

To address these issues, the "Dialog-to-Action" (D2A) framework maps conversational questions into executable logical forms. By incorporating a novel "dialog memory management" component, the model successfully resolves complex linguistic phenomena, significantly outperforming previous memory-network baselines.

The Dialog-to-Action (D2A) Framework

The authors propose a semantic parsing-based approach consisting of three main pillars:

1. Grammar-Guided Decoder

Instead of predicting raw text, the decoder predicts a sequence of "actions" (e.g., find, count, union) based on a predefined, flexible grammar. This ensures that the generated output is always a valid, executable query. By restricting the decoder to a specific grammar, the authors effectively prune the search space, making it feasible to operate over a massive KB.

2. Dialog Memory Management

This is the core innovation of the paper. The model maintains a memory of historical context to resolve missing information in current turns.

Dialog memory management replicating action subsequences

The memory tracks:

  • Entities: Extracted from previous questions and answers.
  • Predicates: Relations mentioned in the previous turn.
  • Action Subsequences: Partial logical forms from previous turns.

When a user asks an elliptical question, they are often asking for the same logic applied to a new entity. D2A can copy the structure of a previous query (Action Subsequence Memory) and simply swap out the subject by instantiating historical elements.

3. Learning from Denotations

To train without "gold" logical forms, the authors use a breadth-first search (BFS) algorithm to find action sequences that lead to the correct answer in the KB. The model is then trained to maximize the likelihood of these successful sequences.

Datasets and Key Results

The model was evaluated on the CSQA (Complex Sequential Question Answering) dataset, which contains 200K dialogs (1.6M turns) over a Wikidata-based KB featuring 12.8M entities.

The D2A model significantly outperformed the previous state-of-the-art baseline, HRED+KVmem (a hierarchical encoder-decoder with key-value memory):

  • Massive Performance Gains: D2A achieved an overall Recall of 64.04% and Precision of 61.76%, compared to the baseline's 18.40% Recall and 6.30% Precision.
  • Ellipsis Handling: In the "Simple Question (Ellipsis)" category, D2A reached 86.58% Recall. Ablation studies showed that removing the "Predicate Memory" caused this to plummet to ~15%, proving that remembering previous relations is vital for resolving omitted context.
  • Co-reference Resolution: For questions involving pronouns, the "Entity Memory" was crucial; removing it dropped accuracy from ~71% to ~38%.

These results highlight a key insight: semantic parsing (converting text to logic) is inherently more powerful for complex reasoning than text-to-text generation. By explicitly manipulating functions and entities, the model can perform set operations that are difficult for standard Seq2Seq models.

Limitations and Future Work

While highly effective, the authors highlight several areas for improvement:

  • Entity Linking: The model struggles when different entities share the same name (surface form). Better word representations incorporating KB context could help.
  • Spurious Programs: The BFS training method sometimes finds "lucky" queries—logical forms that get the right answer for the wrong reason.
  • Error Propagation: Because the model replicates previous action sequences, an error in an early turn can cascade through the entire conversation.
  • Grammar Coverage: Expanding the action set to cover more complex queries (e.g., specific diplomatic relations) without exploding the search space remains a future goal.

Summary updated

Abstract

We present an approach to map utterances in conversation to logical forms, which will be executed on a large-scale knowledge base. To handle enormous ellipsis phenomena in conversation, we introduce dialog memory management to manipulate historical entities, predicates, and logical forms when inferring the logical form of current utterances. Dialog memory management is embodied in a generative model, in which a logical form is interpreted in a top-down manner following a small and flexible grammar. We learn the model from denotations without explicit annotation of logical forms, and evaluate it on a large-scale dataset consisting of 200K dialogs over 12.8M entities. Results verify the benefits of modeling dialog memory, and show that our semantic parsing-based approach outperforms a memory network based encoder-decoder model by a huge margin.
BibTeX
@inproceedings{guo2018dialog,
  author = {Daya Guo and Duyu Tang and Nan Duan and Ming Zhou and Jian Yin},
  title = {{Dialog-to-Action: Conversational Question Answering Over a Large-Scale Knowledge Base}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=HJ-kXKbO-S},
}
Figure