You May Like This Hotel Because ...: Identifying Evidence for Explainable Recommendations

Shin Kanouchi, Masato Neishi, Yuta Hayashibe, Hiroki Ouchi, Naoaki OkazakiAACL/IJCNLP 2020 · OpenReview

TL;DR

This paper tackles the challenge of providing explainable hotel recommendations in conversational AI by identifying and explaining evidence from reviews that supports a user's vague natural language request. They introduce a novel dataset and demonstrate that BERT can effectively find evidence and an LSTM model can generate natural-sounding explanations.

What Problem Does This Paper Address?

In interactive services like hotel booking, users often make vague requests (e.g., "a child-friendly hotel"). To satisfy these requests, human operators don't just recommend a hotel; they explain why it's a good fit, often citing specific features from reviews. This paper addresses the difficulty of automating this process. Existing explainable recommendation systems often rely on predefined features and struggle with natural language inputs, making them unsuitable for conversational agents. The core problem is how to automatically identify relevant pieces of information (evidence) within hotel reviews that justify a recommendation for a vague user request, and then use that evidence to generate a natural language explanation.

Methods

The authors decompose the problem into two main subtasks:

  1. Evidence Identification: This task aims to predict whether a given sentence in a hotel review contains evidence that supports a user's vague request. They frame this as a binary classification problem.
  2. Evidence Explanation: This task involves generating a natural language recommendation sentence that explains why a hotel is suitable, based on a identified evidence sentence and the user's request. This is treated as a sequence-to-sequence generation problem.

For evidence identification, they experiment with logistic regression and BERT. For evidence explanation, they compare a rule-based approach with LSTM and Transformer models.

Datasets

The paper introduces a novel Evidence-based Explanation (EBE) dataset. This dataset is built from hotel reviews, specifically using review titles as a proxy for vague requests. It contains:

  • Vague requests derived from review titles.
  • Evidence sentences from the corresponding review texts that support these requests.
  • Recommendation sentences generated from the evidence.

The dataset is notable for its size (37,280 reviews with annotations) and the diversity of requests (15,767 unique types).

Key Insights and Ideas

  • Decomposition is Key: Breaking down explainable recommendations into evidence identification and explanation simplifies the problem and allows for specialized model development.
  • Vague Requests are Common and Challenging: The dataset highlights the wide variety of natural language requests users make, which are not easily captured by predefined categories.
  • BERT Excels at Evidence Identification: The BERT model significantly outperforms simpler models like logistic regression in identifying relevant evidence sentences, even for requests not seen during training.
  • LSTM is Effective for Explanation Generation: The LSTM model, when provided with evidence, can generate coherent and natural-sounding recommendation sentences, though it still lags behind human-generated explanations.
  • Context Matters: The performance difference between models that use the request versus those that only use the sentence for evidence identification underscores the importance of the user's query in understanding evidence.

Key Results

  • Evidence Identification: The BERT model achieved an F1-score of 79.94 for evidence identification across all categories, a substantial improvement over logistic regression. It also showed good performance on unseen requests, dropping only by 6.44 points.
  • Evidence Explanation:
    • The LSTM-based model achieved a BLEU score of 56.09 when given a gold evidence sentence and request.
  • When the system had to identify the evidence first (pipeline approach), the BLEU score was 45.38.
  • A rule-based model achieved a BLEU score of 50.26, indicating that the task might require less complex rewriting than initially assumed.

Limitations and Future Work

  • Limitations:
    • The authors acknowledge that the rule-based explanation model can produce longer, less natural sentences and may struggle with irrelevant information in evidence sentences.
  • The end-to-end model (directly generating recommendations from review text) performed significantly worse than the pipeline approach, suggesting challenges with long sequences and potentially over-reliance on the "[no-evidence]" token.
  • The model sometimes fails to infer implicit connections between requests and evidence (e.g., associating "hot spring" with "extraordinary experience").
  • Future Work:
    • Developing methods to select the best evidence sentence when multiple candidates exist.
  • Creating a more sophisticated "concierge service" that can recommend hotels with evidence.
  • Improving the end-to-end approach to better handle reviews without explicit evidence.

Summary updated

BibTeX
@inproceedings{kanouchi2020you,
  author = {Shin Kanouchi and Masato Neishi and Yuta Hayashibe and Hiroki Ouchi and Naoaki Okazaki},
  title = {{You May Like This Hotel Because ...: Identifying Evidence for Explainable Recommendations}},
  booktitle = {AACL/IJCNLP},
  year = {2020},
  url = {https://openreview.net/forum?id=7MvxixccL3O},
}
Figure