PRODIGy: a PROfile-based DIalogue Generation dataset

Daniela Occhipinti, Serra Sinem Tekiroglu, Marco GueriniNAACL 2024 · DOI

TL;DR

This paper introduces PRODIGY, a novel dataset for dialogue generation that incorporates rich, multi-faceted character profiles (biography, personality, gender, dialogue style). They demonstrate that leveraging these profiles significantly improves dialogue coherence and consistency, outperforming models trained without them, especially in limited context scenarios.

What Problem Does This Paper Address?

Creating AI dialogue agents that can hold natural, engaging conversations is a major challenge. While current models can generate fluent text, they often struggle with maintaining a consistent persona or "character" throughout a conversation. This leads to responses that can be uninformative, contradictory, or simply "generic," eroding user trust and engagement. Existing datasets for training such agents often rely on simple, dialogue-specific profiles or implicit representations that are hard to collect. This paper addresses the need for a more comprehensive dataset and a deeper understanding of how different aspects of a character's profile impact dialogue generation.

Methods

The core of this work is the creation of the PRODIGY dataset. They started with the Cornell Movie Dialogs Corpus and enriched it with various profile dimensions for each character:

  • Dialogue History: Implicitly captured through character IDs, representing their linguistic style.
  • Personality: Annotated using the Myers-Briggs Type Indicator (MBTI) from the Personality Database (PDB).
  • Gender: Binary labels from the original corpus.
  • Biography: Extracted and summarized from external sources, then refined through human annotation and paraphrased using LLMs (like ChatGPT) to enhance variability.

They then evaluated dialogue generation models (DialoGPT for fine-tuning and GODEL for instruction prompting) using different combinations of these profile dimensions. Experiments included both in-domain (on PRODIGY) and cross-domain (on Persona-Chat) evaluations, as well as automatic metrics (CPPL, Acc@N) and human evaluations.

Datasets

  • Cornell Movie Dialogs Corpus: The source for raw dialogue data.
  • Personality Database (PDB): Used to extract MBTI personality types.
  • Charactour.com, Fandom.com, Wikipedia: Sources for character biographies.
  • Persona-Chat: Used for cross-domain evaluation.
  • PRODIGY Dataset (New): The primary contribution, comprising over 20,000 dialogues with rich, multi-dimensional character profiles.

Key Insights and Ideas

  • Rich Profiles Matter: The paper strongly suggests that moving beyond simple persona descriptions to include detailed biographies and personality traits significantly enhances dialogue generation quality.
  • Context is King, but Profile is Queen: While dialogue context is crucial, profile information becomes especially vital when the dialogue context is sparse, helping the AI maintain a consistent and believable character.
  • Biography is Powerful: The biographical information, when properly processed and presented, proved to be a highly effective dimension for improving dialogue generation, even outperforming simpler dialogue-specific personas in cross-domain generalization.
  • Implicit vs. Explicit: Both implicit (dialogue history) and explicit (MBTI, gender, biography) profile information contribute positively, with combinations often yielding the best results.
  • Human Preference Shifts: While humans initially prefer generic responses (perhaps for their broad applicability), they significantly favor profile-consistent responses when the profile information is made available to them.

Key Results

  • Automatic Metrics: Fine-tuning DialoGPT with profile information consistently improved Conditional Turn Perplexity (CPPL) and Average Accuracy at N (Acc@N) compared to plain dialogue generation. For instance, in inter-character experiments, models with profile information significantly outperformed the baseline.
  • Cross-Domain Generalization: PRODIGY-based models showed better generalization capabilities on the Persona-Chat dataset than models trained on Persona-Chat itself, especially when using biographical information.
  • Human Evaluation: While gold responses were preferred overall, profile-based models showed significant improvement when evaluators were aware of the speaker's profile. The Gender model yielded the most coherent responses, while the Biopar model (using selected biography sentences) produced the most grounded responses. When profile information was explicitly provided to evaluators, the gap between profile-based models and plain dialogue narrowed, and profile-based models were perceived as more appropriate.

Limitations and Future Work

  • Stereotyping: The use of fictional characters from movie scripts might lead to stereotyped roles, potentially limiting the realism of the generated dialogues.
  • Gender Binary: The dataset is limited to binary gender classification due to the source corpus.
  • Gold Response Preference: Human evaluators still showed a strong preference for gold responses, indicating room for further improvement in model performance.

The authors plan to address these limitations in future work, potentially by exploring more diverse character representations and improving model generation to better match human expectations.

Summary updated

BibTeX
@inproceedings{occhipinti2024prodigy,
  author = {Daniela Occhipinti and Serra Sinem Tekiroglu and Marco Guerini},
  title = {{PRODIGy: a PROfile-based DIalogue Generation dataset}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.222},
}
Figure