AmazonQA: A Review-Based Question Answering Task

Mansi Gupta, Nitish Kulkarni, Raghuveer Chanda, Anirudha Rayasam, Zachary C. LiptonIJCAI 2019 · DOI

TL;DR

This paper introduces AmazonQA, a new dataset and task for question answering (QA) where answers must be synthesized from product reviews. They tackle the challenge of providing immediate, informative answers in a real-world e-commerce setting, where questions are often only partially answerable by existing reviews.

What Problem Does This Paper Address?

Online marketplaces like Amazon are flooded with customer questions about products. While these questions can be incredibly valuable for potential buyers, getting an answer can take days, and sometimes questions go unanswered indefinitely. The authors observe that product reviews, which are often more detailed than the Q&A sections, contain a wealth of information that could be used to answer these questions. The core problem is to build QA systems that can effectively leverage these reviews to provide timely and accurate answers, mimicking a helpful community member. This is particularly challenging because questions might not be fully answerable by any single review, requiring synthesis of information.

Methods

The paper proposes a two-stage approach for review-based QA:

  1. Information Retrieval (IR) for Review Selection: Given a question, the system first identifies the most relevant reviews. This is achieved by tokenizing reviews, chunking them into snippets, and ranking these snippets using the BM25 metric for relevance to the question. The top 10 snippets are then passed to the QA system.
  2. "Reading Comprehension" for Answer Synthesis: A QA system then synthesizes an answer from the selected review snippets. The authors explore several baseline models for this, including:
  3. Language Models: These models approximate conditional probabilities of an answer given the question ($P(a|q)$) and given the question and reviews ($P(a|q, R)$). They use an LSTM-based encoder-decoder architecture.
  4. Span-based QA Models: To adapt existing reading comprehension models, they convert the dataset into a span-based format. This involves heuristically identifying answer spans within reviews using metrics like BLEU and ROUGE, and then training models like R-Net.
  5. Answerability Classifier: To handle questions that cannot be answered from reviews, they train a classifier using crowdsourced labels to determine if a question-context pair is answerable.

Datasets

The primary contribution is the AmazonQA dataset, built upon the existing dataset from McAuley and Yang (2016). This new dataset includes:

  • 923k questions
  • 3.6M answers
  • 14M reviews
  • 156k products

Crucially, they added annotations to mark each question-context pair as answerable or unanswerable. They also provide pre-processed review snippets relevant to each question. The dataset is split into training, development, and test sets based on products, not individual QA pairs, to prevent data leakage.

Key Insights and Ideas

  • Reviews are a Rich Source: A significant portion (over half) of customer questions can be at least partially answered using existing product reviews, highlighting the untapped potential of this data.
  • Answerability is Key: Not all questions are answerable from reviews. Developing a robust answerability classifier is crucial for building practical systems that can gracefully handle unanswerable queries.
  • Real-World Data is Messy: The dataset is extracted from actual user interactions, meaning the questions, answers, and reviews are naturally diverse, sometimes ambiguous, and often require synthesizing information from multiple sources. This contrasts with many existing QA datasets that might be more curated or synthetically generated.
  • Heuristics as Strong Baselines: Simple heuristic-based methods (like picking the top-ranked sentence from reviews) perform surprisingly well, even outperforming more complex neural models on certain metrics, suggesting the difficulty of the task and the need for more sophisticated approaches.

Key Results

  • The proposed answerability classifier achieves 72% precision in identifying answerable question-review pairs.
  • Language models trained on the AmazonQA dataset show decreasing perplexity as more context (question and reviews) is provided ($P(a) > P(a|q) > P(a|q, R)$), confirming the utility of both question and review information.
  • Heuristic-based answer generation methods, particularly those using IR or BLEU scores to select sentences, achieve higher scores (e.g., Rouge scores around 60-62) than the span-based neural model (R-Net, ~40 Rouge) on the test set.
  • Human evaluation shows that Amazon users provide answers that are generally better than the heuristic baselines, but the gap is not insurmountable, indicating room for improvement for automated systems.

Limitations and Future Work

  • Span Heuristics for Supervision: The conversion of the dataset to a span-based format for training models like R-Net relies on heuristics, which are imperfect and can lead to noisy supervision.
  • Evaluation Metrics: Standard metrics like BLEU and ROUGE, which focus on token similarity, might not be ideal for evaluating descriptive answers, especially for yes/no questions. The authors suggest a combined metric approach.
  • Future Work: The authors suggest exploring more sophisticated neural architectures for answer generation, improving the answerability classifier, and developing better evaluation metrics. They also highlight the potential for this dataset to be used for learning out-of-domain question answering due to its diverse categories.

Summary updated

BibTeX
@inproceedings{gupta2019amazonqa,
  author = {Mansi Gupta and Nitish Kulkarni and Raghuveer Chanda and Anirudha Rayasam and Zachary C. Lipton},
  title = {{AmazonQA: A Review-Based Question Answering Task}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/694},
}
Figure