ClidSum: A Benchmark Dataset for Cross-Lingual Dialogue Summarization

Jiaan Wang, Fandong Meng, Ziyao Lu, Duo Zheng, Zhixu Li, Jianfeng Qu, Jie Zhou

TL;DR

The authors introduce CLIDSUM, the first large-scale benchmark for Cross-Lingual Dialogue Summarization (XLDS), featuring over 67k dialogues and 112k professional human translations in German and Chinese. They also propose mDIALBART, a model that uses a specialized second stage of pre-training to capture dialogue structures, successfully outperforming traditional pipeline-based summarization methods.

What Problem Does This Paper Address?

Cross-Lingual Summarization (XLS) involves taking a document in one language (e.g., English) and generating a summary in another (e.g., Chinese). While well-studied for structured, single-author documents like news reports, it remains a significant challenge for dialogues.

Dialogues are inherently "messy": they involve multiple speakers, informal language, and complex turn-taking structures. Furthermore, the lack of large-scale, high-quality datasets has hindered the development of end-to-end models for this task. Researchers previously relied on "pipeline" methods—either summarizing first then translating, or vice versa—which are prone to error propagation and high latency.

Methods

The researchers developed mDIALBART, an end-to-end model based on mBART-50. To bridge the gap between general multilingual pre-training and the specific needs of dialogue summarization, they introduced a second stage of pre-training with four distinct objectives:

  1. Action Infilling (AcI): The model learns to reconstruct "who-doing-what" triples within utterances, improving factual consistency.
  2. Utterance Permutation (UP): The model must restore the original order of shuffled dialogue turns, forcing it to learn discourse structure.
  3. Monolingual Dialogue Summarization (MDS): Learning to summarize within the source language.
  4. Machine Translation (MT): Learning the direct mapping between source and target languages.
Figure 2: The second stage of pre-training in mDIALBART. <En> and <Zh> are two language identifiers indicat- ing the input and output languages. [SUM] is a special token to indicate the summarization task.

During fine-tuning, they use a special [SUM] token to trigger the summarization behavior learned during pre-training.

Datasets

The paper introduces CLIDSUM, constructed by taking two popular monolingual dialogue datasets—SAMSum (daily chats) and MediaSum (interview transcripts)—and employing professional translators to create German and Chinese summaries.

Table 1: Statistics of CLIDSUM and previous synthetic XLS datasets. Trans. indicates the translation method (automatic or manual) to construct dataset. Src Lang. and Tgt Lang. denote the source and target languages (En: English, Zh: Chinese or De: German) for each dataset. Documents represents the size of each dataset. Doc. Length, Src Summ Length and Tgt Summ Length show the average length of documents, source summaries and target summaries (word-level for English and German while character-level for Chinese) for each dataset, repectively.

As shown in the table above, CLIDSUM is unique because it is the first large-scale XLS dataset specifically for dialogues and the first to use manual (human) translation rather than automated synthetic methods.

Key Insights and Ideas

  • Dialogue Structure Matters: Standard summarization models often struggle with the "who said what" aspect of conversations. The Utterance Permutation and Action Infilling tasks are critical for helping the model respect the flow of a multi-party conversation.
  • End-to-End > Pipeline: While strong pipelines (like using Google Translate) are competitive, they suffer from "error propagation"—if the summarizer makes a mistake, the translator carries it over (or makes it worse). End-to-end models like mDIALBART learn to handle both tasks simultaneously, leading to more coherent results.
  • The Power of Semi-Supervision: Because cross-lingual data is scarce, the authors show that using large amounts of monolingual dialogue data during the second pre-training stage significantly boosts performance on the final cross-lingual task.

Key Results

The main experiments compared mDIALBART against various pipeline baselines (using models like BART, T5, and PEGASUS combined with MT services).

Table 3: Experimental results on CLIDSUM. The bold and underline denote the best and the second scores, respectively. † represents that the evaluated models utilize the monolingual dialogue-summary pairs provided by the semi-supervised setting. Sum-Trans: Summarize-then-translate, Trans-Sum: Translate-then-summarize.
  • Superiority of mDIALBART: In the semi-supervised setting (row 24), mDIALBART outperformed all baselines across ROUGE-1, ROUGE-2, and ROUGE-L metrics.
  • Data Augmentation (DA): Adding pseudo-cross-lingual samples (row 25) further improved results, achieving the state-of-the-art on the MediaSum40k test set (e.g., 28.7 ROUGE-1 for En⇒Zh).
  • Human Evaluation: Human judges rated mDIALBART higher than pipeline models in terms of grammaticality, informativeness, and conciseness.

Limitations and Future Work

The authors identify several hurdles for future research:

  • Domain Adaptation: mDIALBART performed slightly worse on the SAMSum dataset compared to MediaSum, suggesting that the model's ability to generalize across different dialogue types (e.g., from formal interviews to casual chats) needs improvement.
  • Multiple Topics: Long dialogues often cover several unrelated events. Current models struggle to capture all distinct topics in a single summary.
  • Inference Efficiency: mBART-50 contains parameters for 50 languages. For a specific task like En⇒Zh, many of these parameters are redundant, which slows down inference speed. Future work could involve "pruning" the model for specific language pairs.

Summary updated

Abstract

We present ClidSum, a benchmark dataset for building cross-lingual summarization systems on dialogue documents. It consists of 67k+ dialogue documents from two subsets (i.e., SAMSum and MediaSum) and 112k+ annotated summaries in different target languages. Based on the proposed ClidSum, we introduce two benchmark settings for supervised and semi-supervised scenarios, respectively. We then build various baseline systems in different paradigms (pipeline and end-to-end) and conduct extensive experiments on ClidSum to provide deeper analyses. Furthermore, we propose mDialBART which extends mBART-50 (a multi-lingual BART) via further pre-training. The multiple objectives used in the further pre-training stage help the pre-trained model capture the structural characteristics as well as important content in dialogues and the transformation from source to the target language. Experimental results show the superiority of mDialBART, as an end-to-end model, outperforms strong pipeline models on ClidSum. Finally, we discuss specific challenges that current approaches faced with this task and give multiple promising directions for future research. We have released the dataset and code at https://github.com/krystalan/ClidSum.
BibTeX
@inproceedings{wang2022clidsum,
  author = {Jiaan Wang and Fandong Meng and Ziyao Lu and Duo Zheng and Zhixu Li and Jianfeng Qu and Jie Zhou},
  title = {{ClidSum: A Benchmark Dataset for Cross-Lingual Dialogue Summarization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.526},
}
Figure