Is a Knowledge-based Response Engaging?: An Analysis on Knowledge-Grounded Dialogue with Information Source Annotation

Takashi Kodama, Hirokazu Kiyomaru, Yin Jou Huang, Taro Okahisa, Sadao KurohashiACL 2023

TL;DR

This paper investigates what makes knowledge-grounded dialogue responses engaging. By analyzing human conversations, they found that incorporating "speaker-derived" information (personal opinions, experiences) alongside external knowledge significantly boosts engagement. Current models, however, struggle to generate such engaging responses, highlighting a gap in current training paradigms.

What Problem Does This Paper Address?

Modern dialogue systems are increasingly incorporating external knowledge to provide richer and more informative responses. However, simply reflecting this external knowledge doesn't guarantee engaging conversations. Humans, when conversing, don't just relay facts; they weave in their own perspectives, experiences, and opinions. This "speaker-derived" information is crucial for making dialogue feel natural and engaging. The paper addresses the question of whether this speaker-derived information is indeed key to engaging dialogue and whether current knowledge-grounded dialogue models can effectively generate it.

Methods

The researchers developed an annotation scheme to categorize information within dialogue utterances. They analyzed the Japanese Movie Recommendation Dialogue (JMRD) corpus, annotating entities (nouns, verbs, adjectives) with their information source:

  • Database-derived: Information directly from the provided external knowledge.
  • Speaker-derived: Information stemming from the speaker's own knowledge, experiences, or opinions.
  • Other: Miscellaneous information like greetings.

They then conducted two main analyses:

  1. Human Utterance Analysis: They analyzed the JMRD corpus to understand how humans naturally incorporate speaker-derived information and its correlation with dialogue engagingness, both at the dialogue and utterance levels.
  2. System Utterance Analysis: They trained a standard BART-based response generation model on the JMRD dataset. Using a trained Information Source Classifier, they then analyzed the types of information sources present in the model's generated responses and compared them to human responses.

Datasets

The primary dataset used is the Japanese Movie Recommendation Dialogue (JMRD) corpus. This is a human-to-human dialogue corpus where a recommender suggests movies to a seeker. Each recommender utterance is associated with external movie knowledge. The paper uses this corpus for both annotation and training/evaluation of their models. No new benchmark datasets were introduced.

Key Insights and Ideas

  • Speaker-Derived Information is Key to Engagement: The most significant finding is that utterances and dialogues containing more speaker-derived information are rated as more engaging by humans. This suggests that personal touch and subjective input are vital for making conversations interesting.
  • Models Lag Behind Humans: Current knowledge-grounded dialogue models, even when trained to reflect external knowledge, fail to incorporate speaker-derived information as frequently or effectively as humans do. This indicates a fundamental limitation in current training objectives.
  • Minimizing Perplexity Isn't Enough: The study implies that simply optimizing models to minimize perplexity (a common objective in language generation) is insufficient for achieving engaging, knowledge-grounded dialogue.
  • Knowledge Type Matters for Speaker-Derived Content: The analysis revealed that speaker-derived information drops significantly for longer, more informative knowledge types like "reviews" and "plots." This suggests models might prioritize direct knowledge injection over subjective elaboration for complex information.

Key Results

  • Human Analysis:
    • Dialogues with higher engagingness scores contained a greater proportion of speaker-derived entities.
  • Similarly, at the utterance level, responses with speaker-derived entities received higher engagingness scores (3.31) compared to those without (3.07), a statistically significant difference.
  • System Analysis:
    • The trained response generation model showed a significant increase in database-derived labels (from 27.49% in human predictions to 85.48% in system predictions) and a corresponding decrease in speaker-derived information (from 66.75% to 10.66%).
  • An example showed the system accurately reflecting factual knowledge but missing a nuanced speaker-derived point (e.g., voice actors also being singers) that a human would likely include.

Limitations and Future Work

  • Model Limitations: The paper acknowledges that the standard training approach (minimizing perplexity) is inadequate for generating engaging, speaker-derived content. The model's inability to effectively incorporate speaker-derived information, especially for complex knowledge types, is a key limitation.
  • Future Directions: The authors suggest the need for a novel learning framework that can better capture and generate speaker-derived information. They hope their annotated corpus will serve as a foundation for future research in this area, aiming to develop models that can produce more engaging and human-like dialogue.

Summary updated

BibTeX
@inproceedings{kodama2023is,
  author = {Takashi Kodama and Hirokazu Kiyomaru and Yin Jou Huang and Taro Okahisa and Sadao Kurohashi},
  title = {{Is a Knowledge-based Response Engaging?: An Analysis on Knowledge-Grounded Dialogue with Information Source Annotation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure