TL;DR
The authors introduce CLIDSUM, the first large-scale benchmark for Cross-Lingual Dialogue Summarization (XLDS), featuring over 67k dialogues and 112k professional human translations in German and Chinese. They also propose mDIALBART, a model that uses a specialized second stage of pre-training to capture dialogue structures, successfully outperforming traditional pipeline-based summarization methods.
What Problem Does This Paper Address?
Cross-Lingual Summarization (XLS) involves taking a document in one language (e.g., English) and generating a summary in another (e.g., Chinese). While well-studied for structured, single-author documents like news reports, it remains a significant challenge for dialogues.
Dialogues are inherently "messy": they involve multiple speakers, informal language, and complex turn-taking structures. Furthermore, the lack of large-scale, high-quality datasets has hindered the development of end-to-end models for this task. Researchers previously relied on "pipeline" methods—either summarizing first then translating, or vice versa—which are prone to error propagation and high latency.
Methods
The researchers developed mDIALBART, an end-to-end model based on mBART-50. To bridge the gap between general multilingual pre-training and the specific needs of dialogue summarization, they introduced a second stage of pre-training with four distinct objectives:
- Action Infilling (AcI): The model learns to reconstruct "who-doing-what" triples within utterances, improving factual consistency.
- Utterance Permutation (UP): The model must restore the original order of shuffled dialogue turns, forcing it to learn discourse structure.
- Monolingual Dialogue Summarization (MDS): Learning to summarize within the source language.
- Machine Translation (MT): Learning the direct mapping between source and target languages.
During fine-tuning, they use a special [SUM] token to trigger the summarization behavior learned during pre-training.
Datasets
The paper introduces CLIDSUM, constructed by taking two popular monolingual dialogue datasets—SAMSum (daily chats) and MediaSum (interview transcripts)—and employing professional translators to create German and Chinese summaries.
As shown in the table above, CLIDSUM is unique because it is the first large-scale XLS dataset specifically for dialogues and the first to use manual (human) translation rather than automated synthetic methods.
Key Insights and Ideas
- Dialogue Structure Matters: Standard summarization models often struggle with the "who said what" aspect of conversations. The Utterance Permutation and Action Infilling tasks are critical for helping the model respect the flow of a multi-party conversation.
- End-to-End > Pipeline: While strong pipelines (like using Google Translate) are competitive, they suffer from "error propagation"—if the summarizer makes a mistake, the translator carries it over (or makes it worse). End-to-end models like mDIALBART learn to handle both tasks simultaneously, leading to more coherent results.
- The Power of Semi-Supervision: Because cross-lingual data is scarce, the authors show that using large amounts of monolingual dialogue data during the second pre-training stage significantly boosts performance on the final cross-lingual task.
Key Results
The main experiments compared mDIALBART against various pipeline baselines (using models like BART, T5, and PEGASUS combined with MT services).
- Superiority of mDIALBART: In the semi-supervised setting (row 24), mDIALBART outperformed all baselines across ROUGE-1, ROUGE-2, and ROUGE-L metrics.
- Data Augmentation (DA): Adding pseudo-cross-lingual samples (row 25) further improved results, achieving the state-of-the-art on the MediaSum40k test set (e.g., 28.7 ROUGE-1 for En⇒Zh).
- Human Evaluation: Human judges rated mDIALBART higher than pipeline models in terms of grammaticality, informativeness, and conciseness.
Limitations and Future Work
The authors identify several hurdles for future research:
- Domain Adaptation: mDIALBART performed slightly worse on the SAMSum dataset compared to MediaSum, suggesting that the model's ability to generalize across different dialogue types (e.g., from formal interviews to casual chats) needs improvement.
- Multiple Topics: Long dialogues often cover several unrelated events. Current models struggle to capture all distinct topics in a single summary.
- Inference Efficiency: mBART-50 contains parameters for 50 languages. For a specific task like En⇒Zh, many of these parameters are redundant, which slows down inference speed. Future work could involve "pruning" the model for specific language pairs.
Summary updated