TL;DR
This paper introduces GENQA, an end-to-end neural network that generates answers to factoid questions by querying a knowledge base. It combines sequence-to-sequence learning with a novel "enquirer" module to retrieve relevant facts, outperforming previous embedding-based and dialogue models.
What Problem Does This Paper Address?
Traditional question answering (QA) systems often rely on complex pipelines involving semantic parsing and rule-based systems, requiring significant linguistic expertise. While end-to-end neural models have shown promise in generating natural language responses, they struggle to store and access the vast, discrete knowledge required for real-world QA. This paper addresses the challenge of building a purely generative QA system that can effectively leverage external knowledge bases to answer factoid questions accurately and naturally.
Methods
GENQA employs an encoder-decoder framework, similar to neural machine translation. The core innovation lies in its architecture, which includes:
- Interpreter: This module encodes the input question into a representation stored in short-term memory. It uses a bidirectional RNN (specifically GRU) for this purpose.
- Enquirer: This novel component interacts with an external knowledge base (KB). It first retrieves a set of candidate facts (triples) relevant to the question and then scores these facts to identify the most pertinent ones. The paper proposes two implementations for the Enquirer: a Bilinear Model and a CNN-based Matching Model.
- Answerer: This module, also an RNN, generates the answer sentence. It leverages both the question representation from short-term memory and the retrieved knowledge from the Enquirer. A key feature of the Answerer is a mixture model that allows it to generate words from either a common vocabulary or a dynamic KB vocabulary (i.e., the objects of the retrieved triples). This enables the model to seamlessly blend common words with specific factual entities.
The entire GENQA model is trained end-to-end, optimizing the likelihood of observed question-answer pairs.
Datasets
The authors introduce a new dataset for generative QA. This dataset was constructed by:
- Building a Knowledge Base: Mining facts (triples of subject, predicate, object) from three Chinese encyclopedia websites.
- Collecting Question-Answer Pairs: Extracting pairs from two Chinese community QA sites.
The training and test data were created by "grounding" the QA pairs with triples from the KB. This process resulted in approximately 720K instances (question, answer, triple) for training and testing, with a focus on simple factoid questions where each QA pair is associated with a single fact. The dataset is made publicly available.
Key Insights and Ideas
- Bridging Generation and Retrieval: The paper successfully demonstrates that a purely generative model can be enhanced by explicitly incorporating a retrieval mechanism (the Enquirer) to access external knowledge. This overcomes the limitation of neural networks in storing vast amounts of discrete knowledge.
- End-to-End Training for Knowledge Integration: By training the entire system end-to-end, GENQA learns to jointly optimize the interpretation of questions, the retrieval of relevant facts, and the generation of natural language answers. This is crucial for effectively integrating knowledge into the generation process.
- Mixture Model for Answer Generation: The Answerer's mixture model for word generation is a critical innovation. It allows the model to dynamically decide whether to use a common word or a specific KB-word, leading to more natural and factually accurate answers.
- Generalization to Unseen Facts: The dataset partitioning strategy, where test data contains triples not seen during training, highlights the model's ability to generalize to new facts, a crucial aspect for real-world applicability.
Key Results
- Outperforms Baselines: GENQA (specifically the GENQACNN variant) achieved the highest accuracy on the test set, outperforming a retrieval-based QA model (36%), a Neural Responding Machine (NRM, 19%), and an embedding-based QA model (45%). GENQACNN reached an accuracy of 52%.
- Effective Fact Integration: The model demonstrated the ability to generate correct answers even for questions involving new facts not seen during training. For instance, it correctly answered "He plays for Spain" for a question about a Spanish athlete.
- Fluency: The paper notes no significant differences in fluency between NRM and GENQA, suggesting that sequence generation models generally produce grammatically correct patterns.
Limitations and Future Work
- Focus on Simple Factoid Questions: The current GENQA model is designed for simple factoid questions, where each QA pair corresponds to a single fact. Extending it to handle more complex questions requiring multi-hop reasoning or synthesis of information from multiple facts is a natural next step.
- Potential for Improper Common Words: While GENQA excels at blending KB-words, the paper acknowledges that in some cases, the surrounding common words might be improper or incorrect. Roughly 8% of correctly generated answers had such issues.
- Short Answers: The model tends to generate short answers, partly due to the nature of factoid questions on community QA sites and partly due to the beam-searching strategy that favors shorter sequences.
- Future Directions: The authors suggest exploring more complex reasoning capabilities and improving the handling of common word generation to further enhance the model's performance and naturalness.
Summary updated