A Large-Scale Dataset for Empathetic Response Generation

Anuradha Welivita, Yubo Xie, Pearl PuEMNLP 2021 · DOI

TL;DR

This paper introduces EDOS, a massive new dataset of 1 million movie dialogues annotated with 32 fine-grained emotions and 8 empathetic response intents. EDOS aims to overcome the limitations of existing smaller datasets, enabling the development of more sophisticated empathetic AI chatbots.

What Problem Does This Paper Address?

Developing AI chatbots that can respond empathetically to human emotions is a significant challenge. While existing datasets exist, they are often too small to train robust models, and they lack the detailed annotation of subtle emotions and specific response strategies (like "sympathizing" or "questioning") that are crucial for nuanced empathetic conversations. Manually annotating such data is prohibitively expensive and time-consuming, hindering progress in this area.

Methods

The authors tackle this by creating a "silver standard" dataset, meaning it's automatically annotated rather than purely human-annotated. Their pipeline involves:

  1. Dialogue Curation: They start with a massive corpus of movie subtitles (OpenSubtitles), which they segment into dialogues and clean extensively.
  2. Weak Labeling: A BERT-based classifier, trained on a smaller, human-annotated dataset (EmpatheticDialogues), is used to predict emotions and intents for the movie dialogues.
  3. Human Annotation (Seed): A small subset of dialogues (9K) is manually annotated by crowd workers to create a "seed" dataset.
  4. Data Augmentation & Strong Labeler: Techniques like sentence embeddings (SBERT) are used to find semantically similar dialogues, effectively augmenting the seed data. This augmented data is then used to train a more accurate "strong labeler" model.
  5. Automatic Annotation: The strong labeler is then used to automatically annotate the vast majority of the 1 million dialogues in EDOS.

Datasets

  • OpenSubtitles 2018: The source corpus for raw movie dialogues.
  • EmpatheticDialogues: A smaller, existing gold-standard dataset used for training the initial weak labeler and for comparison.
  • EDOS (Emotional Dialogues in OpenSubtitles): The newly introduced large-scale dataset, containing 1 million dialogues annotated with 32 emotions and 8 empathetic response intents.

Key Insights and Ideas

  • The Power of Scale: The sheer size of EDOS (1 million dialogues) is a major leap forward, addressing the critical need for large datasets in training complex NLP models.
  • Beyond Basic Emotions: Existing datasets often rely on coarse emotion categories. EDOS's 32 fine-grained emotions allow for a much more nuanced understanding of human emotional expression.
  • Intent Matters: Simply recognizing an emotion isn't enough; understanding the intent behind a response (e.g., to question, to sympathize, to encourage) is vital for generating appropriate empathetic replies. EDOS's inclusion of 8 specific intents is a key contribution.
  • Silver Standard Viability: The paper demonstrates that a carefully constructed pipeline of weak and strong labeling, combined with data augmentation, can produce a high-quality, large-scale dataset that rivals the utility of purely human-annotated ones for certain tasks.
  • Movie Dialogues as a Rich Source: Movie subtitles offer a vast and diverse source of natural, emotionally rich conversations that can be mined for training empathetic AI.

Key Results

  • EDOS Dataset: A dataset of 1 million emotional dialogues, annotated with 32 fine-grained emotions, 8 empathetic response intents, and a Neutral category. This is approximately 40 times larger than the EmpatheticDialogues dataset.
  • Dialogue Emotion Classifier Performance: The authors developed a dialogue emotion classifier with a macro F1-score of 63.86% and an accuracy of 65.00%, comparable to state-of-the-art classifiers.
  • Model Performance: When fine-tuned on EDOS, a transformer model achieved better diversity metrics (Distinct-1 and -2 scores) compared to models trained on smaller datasets, even in zero-shot settings.

Limitations and Future Work

  • English-Only Bias: The dataset is exclusively in English, potentially perpetuating an English bias in NLP systems.
  • Risk of Inappropriate Content: Despite efforts to remove profanity, the inherent nature of movie dialogues means there's a risk of generating inappropriate or biased responses, especially with end-to-end models.
  • Future Work: The authors plan to use EDOS for further experiments in controllable and interpretable response generation, leveraging the fine-grained emotion and intent labels to condition chatbot behavior. They also intend to use it for training advanced dialogue emotion classifiers.

Summary updated

BibTeX
@inproceedings{welivita2021large,
  author = {Anuradha Welivita and Yubo Xie and Pearl Pu},
  title = {{A Large-Scale Dataset for Empathetic Response Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.96},
}
Figure