Toward Data-Driven Tutorial Question Answering with Deep Learning Conversational Models

Mayank Kulkarni, Kristy BoyerNAACL 2018

TL;DR

This paper explores using deep learning to build a conversational AI tutor for Java programming, trained on a massive dataset of questions and answers from Stack Overflow. They investigate retrieval-based, generative, and hybrid models, finding that a hybrid approach shows the most promise for generating helpful, context-aware programming assistance.

What Problem Does This Paper Address?

Creating effective AI-powered educational tools, particularly for complex subjects like programming, faces a significant hurdle: the scarcity of domain-specific training data. While general conversational AI has made strides, building systems that can provide targeted, helpful answers to programming questions in a tutorial setting is challenging. Traditional methods often require extensive manual effort to create question-answer pairs, which is impractical for large-scale educational applications. This paper aims to bridge this gap by leveraging the vast, publicly available data from community-based question-answering forums like Stack Overflow to train deep learning models for Java programming assistance.

Methods

The researchers explored three main deep learning approaches:

  • Retrieval-Based Models: These models work by finding the most relevant answer from a predefined set of existing answers. They employed a Dual Encoder architecture (Siamese network), specifically a Bidirectional LSTM, to learn semantic representations of questions, descriptions, and answers. This allowed them to compare the similarity between a user's query and potential answers. They experimented with two variations:
  • Description & Answer Dual Encoder (DADE): Focused on matching descriptions to answers.
  • Question & Description Dual Encoder (QDDE): Focused on matching questions to descriptions.
  • Generative Models: Unlike retrieval models, these models generate new answers from scratch. They utilized a Sequence-to-Sequence (Seq2Seq) model, a standard architecture for tasks like machine translation, framing the problem as translating a description into a Java answer. This model uses an encoder-decoder structure with Bidirectional LSTMs.
  • Hybrid Model: This approach combines the strengths of both retrieval and generative methods. The proposed hybrid model first uses the QDDE to find the most appropriate description for a user's question and then feeds this description into the Seq2Seq model to generate a tailored answer. This leverages the ability of QDDE to understand the question's intent and the Seq2Seq model's capacity to generate novel, contextually relevant code snippets.

Datasets

The primary dataset for this research was derived from Stack Overflow, a popular community-based question-answering website. The authors collected a dataset of approximately 106,386 question-description-answer triplets specifically for Java programming questions. This dataset was created by filtering the Stack Exchange data dump to include only Java-related questions, excluding those with code snippets in the main text, and ensuring that questions had at least one answer. They also performed significant data cleaning, removing outliers and normalizing sentence lengths. This newly curated dataset is made available for public use.

Key Insights and Ideas

  • Community Q&A as a Goldmine for AI Tutors: Stack Overflow, with its vast amount of user-generated content, provides a rich and readily available source for training domain-specific AI tutors, bypassing the need for expensive manual data creation.
  • Hybrid Models Offer the Best of Both Worlds: Combining retrieval (to find the best context) and generation (to create a specific answer) proved more effective than either approach alone. This suggests that understanding the user's intent (retrieval) and then crafting a precise response (generation) is key.
  • The Challenge of Code and Natural Language: Answers in the dataset often intersperse natural language explanations with Java code. This presents a unique challenge for word embedding models, as they need to capture the semantics of both.
  • Seq2Seq Models Can Generate Coherent Code Snippets: The generative Seq2Seq models, while sometimes producing short or incomplete answers, demonstrated an ability to combine English explanations with Java code, even generating syntactically correct code snippets.

Key Results

  • Retrieval Models (Dual Encoders): Achieved high recall rates, indicating they could effectively identify relevant answers from a set. The QDDE model, in particular, showed strong performance in recall@5, suggesting it could narrow down to a good answer within a group of ten.
  • Generative Models (Seq2Seq): While not achieving the same recall as retrieval models, these models showed promise in generating coherent responses that combined English and Java code.
  • Hybrid Model: This approach demonstrated the most promising results, leveraging the strengths of both retrieval and generation. Qualitative analysis showed that the hybrid model could generate relevant code snippets and explanations, even for questions not seen during training. For example, when asked "How can we create an integer array in java," the hybrid model suggested using ArrayList and generated code demonstrating its usage.

Limitations and Future Work

The authors acknowledge that while their models can generate coherent responses, they sometimes produce answers that are small or don't fully address the question. They also note that the dataset, while large, still has limitations in terms of the variety of questions and the complexity of the programming concepts covered.

Future work could involve:

  • Jointly training the description selection and answer generation components of the hybrid model.
  • Framing the task as a machine comprehension problem, utilizing the entire answer set as context to potentially leverage memory network architectures.
  • Exploring adversarial training to further improve the quality and appropriateness of generated responses.
  • Investigating the use of transfer learning more deeply, as current results suggest it's not yet yielding optimal outcomes.

Summary updated

BibTeX
@inproceedings{kulkarni2018toward,
  author = {Mayank Kulkarni and Kristy Boyer},
  title = {{Toward Data-Driven Tutorial Question Answering with Deep Learning Conversational Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
}
Figure