From spoken dialogue to formal summary: An utterance rewriting for dialogue summarization

Yue Fang, Hainan Zhang, Hongshen Chen, Zhuoye Ding, Bo Long, Yanyan Lan, Yanquan ZhouNAACL 2022 · OpenReview

Abstract

Due to the dialogue characteristics of unstructured contexts and multi-parties with first-person perspective, many successful text summarization works have failed when dealing with dialogue summarization. In dialogue summarization task, the input dialogue is usually spoken style with ellipsis and co-references but the output summaries are more formal and complete. Therefore, the dialogue summarization model should be able to complete the ellipsis content and co-reference information and then produce a suitable summary accordingly. However, the current state-of-the-art models pay more attention on the topic or structure of summary, rather than the consistency of dialogue summary with its input dialogue context, which may suffer from the personal and logical inconsistency problem. In this paper, we propose a new model, named ReWriteSum, to tackle this problem. Firstly, an utterance rewriter is conducted to complete the ellipsis content of dialogue content and then obtain the rewriting utterances. Then, the co-reference data augmentation mechanism is utilized to replace the referential person name with its specific name to enhance the personal information. Finally, the rewriting utterances and the co-reference replacement data are used in the standard BART model. Experimental results on both SAMSum and DialSum datasets show that our ReWriteSum significantly outperforms baseline models, in terms of both metric-based and human evaluations. Further analysis on multi-speakers also shows that ReWriteSum can obtain relatively higher improvement with more speakers, validating the correctness and property of ReWriteSum.

Introduction

Summarizing dialogues presents unique challenges compared to standard document summarization. Human conversation is inherently informal, multi-party, and relies heavily on "linguistic shortcuts" such as ellipsis (omitting obvious words) and co-reference (using pronouns like "he" or "it"). While large language models like BART are fluent, they often struggle with logical consistency in these contexts, sometimes attributing actions to the wrong speaker or losing track of the subject in fragmented exchanges.

Methodology

To address these issues, we propose ReWriteSum, a framework that "cleans up" dialogue before summarization. The architecture consists of three primary components:

  1. Incomplete Utterance Rewriting (IUR): This module restores missing information by predicting word-level edits (substitute, insert, or none). It utilizes a BiLSTM context layer and a UNet-inspired segmentation layer to capture both local and global dialogue history.
  2. Co-reference Data Augmentation (CDA): This stage identifies pronoun chains and replaces ambiguous references with specific names (e.g., replacing "he" with "Jerry"). This strengthens the link between entities and their actions.
  3. Summarization: The processed, formal-style utterances are fed into a fine-tuned BART-large model to generate the final summary.
Overview of the ReWriteSum architecture including data augmentation and edit matrix generation.

Experimental Results

We evaluated ReWriteSum on two major benchmarks: SAMSum (informal messenger chats) and DialSum (structured spoken dialogues).

Quantitative Performance

ReWriteSum achieved state-of-the-art results on both datasets. On SAMSum, it reached a ROUGE-L score of 50.1, outperforming standard BART-large (47.1). On DialSum, the improvement was more modest (32.1 vs 31.2), likely because the source text in DialSum is naturally more complete than in SAMSum.

Multi-Speaker Analysis

A key finding is that ReWriteSum's performance advantage over baseline models increases as the number of participants in a dialogue grows. This demonstrates the model's superior ability to untangle complex webs of references in group conversations.

ROUGE-L performance comparison between ReWriteSum and BART across different numbers of participants.

Human Evaluation

Human evaluators preferred ReWriteSum summaries in over 50% of head-to-head comparisons. Notably, the model reduced "Missing Information" errors by over 60% compared to the baseline, confirming that the rewriting step successfully preserves factual consistency.

Conclusion

ReWriteSum demonstrates that preprocessing dialogue to resolve ellipsis and co-reference is a highly effective strategy for improving summarization quality. By transforming informal spoken style into a more complete, formal representation, we significantly reduce logical inconsistencies and improve performance in complex, multi-party scenarios.

Summary updated

Abstract

Due to the dialogue characteristics of unstructured contexts and multi-parties with first-person perspective, many successful text summarization works have failed when dealing with dialogue summarization. In dialogue summarization task, the input dialogue is usually spoken style with ellipsis and co-references but the output summaries are more formal and complete. Therefore, the dialogue summarization model should be able to complete the ellipsis content and co-reference information and then produce a suitable summary accordingly. However, the current state-of-the-art models pay more attention on the topic or structure of summary, rather than the consistency of dialogue summary with its input dialogue context, which may suffer from the personal and logical inconsistency problem. In this paper, we propose a new model, named ReWriteSum, to tackle this problem. Firstly, an utterance rewriter is conducted to complete the ellipsis content of dialogue content and then obtain the rewriting utterances. Then, the co-reference data augmentation mechanism is utilized to replace the referential person name with its specific name to enhance the personal information. Finally, the rewriting utterances and the co-reference replacement data are used in the standard BART model. Experimental results on both SAMSum and DialSum datasets show that our ReWriteSum significantly outperforms baseline models, in terms of both metric-based and human evaluations. Further analysis on multi-speakers also shows that ReWriteSum can obtain relatively higher improvement with more speakers, validating the correctness and property of ReWriteSum.
BibTeX
@inproceedings{fang2022from,
  author = {Yue Fang and Hainan Zhang and Hongshen Chen and Zhuoye Ding and Bo Long and Yanyan Lan and Yanquan Zhou},
  title = {{From spoken dialogue to formal summary: An utterance rewriting for dialogue summarization}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=ZeC9zoWKD6},
}
Figure