GENDEX: Generative Data Augmentation Strategy Leveraging External Data for Abstractive Dialogue Summarization

Sangwon Park, Hongseok Choi, Dongha Choi, Hyunju LeeACL 2024 · OpenReview

Abstract

With the proliferation of digital communication, dialogue summarization has become increasingly important. However, it still faces a shortage of data. To address this issue, we developed Generative Data Augmentation Strategy Leveraging External Data for Abstractive Dialogue Summarization (GENDEX), which is based on the hypothetical foundation that texts containing people and their interpersonal interactions can potentially serve as summaries of corresponding dialogues. We filter short texts containing people and resolve coreferences for better contextual analysis. We then identify the semantic roles of words within the texts and filter them based on the patterns observed in the dialogue summarization datasets. Using these texts, we generate synthetic dialogues through a controlled generation method. To better leverage the augmented data, we utilize noise-tolerant training to fine-tune the summarization model. The experimental results demonstrate the effectiveness of our proposed method, showing its robust performance, generalizability, and scalability. Moreover, performance improvements by GENDEX were observed regardless of complexity of dialogues. The code is available at https://github.com/DMCB-GIST/GENDEX.

Introduction

Dialogue summarization is a critical task in natural language processing, yet it suffers from a chronic shortage of high-quality training data. Unlike document summarization, where news headlines often serve as natural summaries, dialogues are structurally complex, turn-based, and frequently contain private information, making manual annotation expensive and difficult. Existing data augmentation techniques, such as word-level perturbations, often fail to introduce sufficient semantic diversity. Furthermore, direct transfer learning from document-to-dialogue domains is hampered by the significant structural gap between narrative text and conversational logs. To overcome these challenges, we propose GENDEX, a novel strategy that leverages abundant external narrative data to synthesize high-quality dialogue-summary pairs.

Methodology

The GENDEX pipeline operates on the "reverse" hypothesis: narrative texts describing interpersonal interactions can be treated as "seed summaries" to generate corresponding synthetic dialogues. The process consists of three primary phases:

1. Source Data Filtering

We identify potential summaries from out-of-domain (OOD) datasets by applying several filters:

  • NER Filtering: Selecting texts with at least two distinct person names.
  • Coreference Resolution: Clarifying context by replacing pronouns with specific names.
  • Semantic Role Labeling (SRL): Extracting predicate-argument patterns (e.g., "A helps B") to ensure the text describes an interaction, matching patterns found in real dialogue summaries.

2. Controlled Dialogue Generation

The filtered texts are used as prompts for a pre-trained DialoGPT model. To ensure coherence and structural integrity, we employ five control parameters: the summary itself, conversation context, number of remaining turns, the next speaker, and turn length.

3. Noise-tolerant Training

Synthetic data is inherently noisier than human-labeled data. We employ a two-stage training approach: first, pre-training the summarization model on synthetic data with early stopping based on a clean validation set, followed by fine-tuning on the original high-quality dataset.

Overview of the GENDEX pipeline, illustrating the flow from source data filtering to noise-tolerant training.

Experimental Setup

We evaluated GENDEX across three major benchmarks: SAMSum (messenger-style), TweetSumm (customer service), and DialogSum (spoken daily life). External source data was drawn from BookCorpus, Wikipedia (Person-masked), and ROCStories. The strategy was tested across various architectures, including BART, T5, DialogLED, and PEGASUS.

Results and Discussion

GENDEX achieves state-of-the-art performance, significantly outperforming baseline models and previous augmentation methods like CODA and SGA. For instance, on the SAMSum dataset, GENDEX improved the BART-base ROUGE-1 score from 45.80 to 47.83.

Low-Resource Performance and Robustness

A key strength of GENDEX is its effectiveness in low-resource scenarios. When training data is limited to 1% or 5% of the original set, GENDEX provides a substantial performance boost, making it ideal for niche domains where data is extremely scarce.

ROUGE-2 performance comparison between BART and GENDEX across varying percentages of training data.

Qualitative analysis indicates that GENDEX-trained models produce summaries with higher relevance and coherence. Furthermore, the model remains robust as dialogue complexity increases, maintaining performance gains even as the number of participants and turns in the conversation grows.

Conclusion

GENDEX introduces a scalable and effective method for dialogue summarization data augmentation by leveraging the semantic richness of external narrative text. While the pipeline currently depends on the accuracy of upstream NLP tools and is focused on English, it demonstrates significant potential for improving abstractive summarization in data-scarce environments. Future work will explore multilingual applications and further refinement of the interaction filtering process.

Summary updated

Abstract

With the proliferation of digital communication, dialogue summarization has become increasingly important. However, it still faces a shortage of data. To address this issue, we developed **Gen**erative **D**ata Augmentation Strategy Leveraging **Ex**ternal Data for Abstractive Dialogue Summarization (**GENDEX**), which is based on the hypothetical foundation that texts containing people and their interpersonal interactions can potentially serve as summaries of corresponding dialogues. We filter short texts containing people and resolve coreferences for better contextual analysis. We then identify the semantic roles of words within the texts and filter them based on the patterns observed in the dialogue summarization datasets. Using these texts, we generate synthetic dialogues through a controlled generation method. To better leverage the augmented data, we utilize noise-tolerant training to fine-tune the summarization model. The experimental results demonstrate the effectiveness of our proposed method, showing its robust performance, generalizability, and scalability. Moreover, performance improvements by *GENDEX* were observed regardless of complexity of dialogues. The code is available at https://github.com/DMCB-GIST/GENDEX.
BibTeX
@inproceedings{park2024gendex,
  author = {Sangwon Park and Hongseok Choi and Dongha Choi and Hyunju Lee},
  title = {{GENDEX: Generative Data Augmentation Strategy Leveraging External Data for Abstractive Dialogue Summarization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=AsL0W9wcKf},
}
Figure