Construction of Hierarchical Structured Knowledge-based Recommendation Dialogue Dataset and Dialogue System

Takashi Kodama, Ribeka Tanaka, Sadao KurohashiACL 2022

TL;DR

This paper introduces JMRD, the first Japanese dialogue dataset for movie recommendations that pairs dialogues with hierarchically structured external knowledge. They also propose a novel dialogue system that leverages this structured knowledge to improve recommendation quality, outperforming baselines in knowledge selection.

What Problem Does This Paper Address?

Recommending items, especially movies, through dialogue requires systems to go beyond simple preference elicitation and provide in-depth information. This necessitates the use of external knowledge. While many knowledge-grounded dialogue datasets exist, they often rely on plain text or flat knowledge graphs. The authors argue that hierarchically structured knowledge, which organizes information like movie titles, casts, reviews, and plots in a structured manner, is crucial for more informative and appropriate recommendations. However, no existing dataset provides this type of structured knowledge specifically for recommendation dialogues, particularly in Japanese. This gap hinders the development of more sophisticated recommendation dialogue systems.

Methods

The core of the paper's contribution lies in both the dataset construction and the proposed dialogue system.

Dataset Construction (JMRD): The Japanese Movie Recommendation Dialogue (JMRD) dataset was created by crowdsourcing. Workers were tasked with engaging in dialogues where one person (recommender) recommends a movie to another (seeker). Crucially, recommenders had access to hierarchically structured knowledge about movies (title, year, director, cast, genre, review, plot) and were instructed to use it as much as possible, annotating which knowledge items they used for each utterance. This allows for direct association between dialogue turns and specific knowledge. The dialogues are relatively long, averaging 23 turns, and focus on providing detailed information about a single movie.

Dialogue System: The proposed dialogue system is an end-to-end model with three main components:

  1. Encoding Layer: This layer uses BERT to encode the dialogue context (seeker's and recommender's previous utterances), knowledge types, and knowledge contents. It also encodes the history of knowledge types and contents used in previous turns.
  2. Knowledge Selection Layer: This layer leverages transformer encoders to process the knowledge type and content histories. It then uses an attention mechanism to select the most relevant piece of knowledge based on the current dialogue context and the learned histories. This is a key innovation, as it explicitly models the importance of when knowledge is used.
  3. Decoding Layer: A transformer decoder generates the response, conditioned on the dialogue context and the selected knowledge content. The system also incorporates a copy mechanism to facilitate the generation of knowledge-based words.

Datasets

  • Japanese Movie Recommendation Dialogue (JMRD): This is the newly introduced dataset. It comprises approximately 5,200 dialogues, with an average of 23 turns per dialogue. The dialogues are paired with hierarchically structured external knowledge about movies.
  • Existing Datasets (for context): The paper references several existing knowledge-grounded dialogue datasets like Dinan et al. (2019) (Wizard of Wikipedia) and Liu et al. (2020) (DuRecDial), highlighting how JMRD differs by using hierarchically structured knowledge.

Key Insights and Ideas

  • Hierarchical Knowledge is Key: The paper strongly advocates for the importance of hierarchically structured knowledge in recommendation dialogues. This structure allows for more nuanced and context-aware selection of information compared to flat knowledge graphs or plain text.
  • Dialogue History Matters: Explicitly modeling the history of knowledge types and contents used in previous turns significantly improves the system's ability to select appropriate knowledge. This suggests that human recommenders don't just pick knowledge randomly but follow a strategy influenced by what they've already said.
  • Longer Dialogues Enable Deeper Recommendations: The JMRD dataset's average of 23 turns per dialogue allows for more in-depth information sharing, moving beyond simple recommendations to detailed explanations.
  • Japanese Language Focus: The creation of a Japanese dialogue dataset addresses a specific linguistic gap in the field of knowledge-grounded dialogue systems.

Key Results

  • Knowledge Selection Accuracy: The proposed model achieved the highest accuracy (49.72%) in selecting appropriate knowledge compared to baselines like Random (4.18%), TMN BERT (48.81%), and TMN BERT+KH (48.66%). This demonstrates the effectiveness of their hierarchical knowledge selection mechanism.
  • Response Reproducibility: The proposed model did not perform as well in terms of reproducing target responses (BLEUtgt-4: 20.78%), which the authors acknowledge is a known issue and potentially an inappropriate metric for dialogue evaluation.
  • Knowledge Reflection: The model showed strong performance in knowledge reflection (Jaccard: 39.35%, BLEUknow-4: 25.88%), indicating that the selected knowledge is better integrated into the generated responses.
  • User Study: Post-task questionnaires revealed that participants enjoyed the dialogues, found them to be of appropriate length, and confirmed that the dialogues achieved their purpose of movie recommendation.

Limitations and Future Work

  • Response Generation Hallucination: The paper acknowledges that the generated utterances can sometimes not reflect the selected knowledge or contain inconsistent information (hallucination). This is identified as a significant area for future work.
  • Single Knowledge Item Selection: For simplicity, the current study uses only one knowledge item per utterance, even when multiple might be relevant. Future work could explore selecting and utilizing multiple knowledge items simultaneously.
  • Limited Scope of Knowledge: While the dataset includes various knowledge types, future work could explore incorporating even richer or more diverse forms of knowledge.
  • Evaluation Metrics: The authors note that direct response reproducibility metrics might not be ideal for dialogue evaluation and suggest further research into more appropriate evaluation methods.

Summary updated

BibTeX
@inproceedings{kodama2022construction,
  author = {Takashi Kodama and Ribeka Tanaka and Sadao Kurohashi},
  title = {{Construction of Hierarchical Structured Knowledge-based Recommendation Dialogue Dataset and Dialogue System}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure