Abstract
In this paper, we focus on improving the quality of the summary generated by neural abstractive dialogue summarization systems. Even though pre-trained language models generate well-constructed and promising results, it is still challenging to summarize the conversation of multiple participants since the summary should include a description of the overall situation and the actions of each speaker. This paper proposes self-supervised strategies for speaker-focused post-correction in abstractive dialogue summarization. Specifically, our model first discriminates which type of speaker correction is required in a draft summary and then generates a revised summary according to the required type. Experimental results show that our proposed method adequately corrects the draft summaries, and the revised summaries are significantly improved in both quantitative and qualitative evaluations.
1. Introduction
While modern abstractive summarization models like BART and UniLM produce fluent and grammatically correct summaries, they frequently struggle with factual consistency in multi-party dialogues. A common failure mode is the "identity crisis," where the model correctly captures the gist of a conversation but attributes actions to the wrong participants. Human evaluation of standard BART-generated summaries reveals that nearly half of all incorrect samples contain "speaker-related errors," such as swapping names or omitting key participants. This paper addresses this gap by introducing a specialized post-correction pipeline designed to identify and rectify speaker-focused inaccuracies.
2. Methodology
The proposed framework utilizes a two-stage "detect-then-correct" approach to refine draft summaries.
2.1 Self-Supervised Data Generation
To train the system without the need for expensive human-annotated error data, the authors developed a self-supervised strategy. They took high-quality reference summaries and intentionally corrupted them using three distinct operations: Insertion (adding an extra name), Deletion (removing a necessary name), and Replacement (swapping one speaker's name for another).
2.2 Correction Pipeline
The architecture consists of two primary components:
- Speaker Correction Type Discriminator (SCTD): This module analyzes the dialogue context alongside the draft summary to predict the error state: None, Insert, Delete, or Replace.
- Speaker-focused Correction Generator (SCG): Built on BART-large, this module generates a revised summary. It utilizes a specific control token (e.g.,
<REP>) based on the discriminator's output to focus its attention on the identified speaker error. - Auxiliary Speaker Generator: To provide an inductive bias, the model is also trained to list all speakers present in the reference summary, forcing the system to explicitly track the "cast of characters" involved in the dialogue.
3. Experimental Results
The model was evaluated primarily on the SAMSum dataset, a benchmark for messenger-style dialogue summarization.
3.1 Quantitative Performance
The post-correction framework demonstrated consistent improvements across all ROUGE metrics. For BART-base, the system achieved a notable absolute improvement of 2.7% in ROUGE-2. The discriminator (SCTD) proved highly effective, reaching a Micro-F1 score of 93.89% in identifying error types.
3.2 Human Evaluation
Qualitative assessments confirmed the quantitative gains. In head-to-head comparisons, human evaluators preferred the corrected summaries over the original drafts in 60-61% of cases for BART-base and 47-54% for BART-large. This suggests that the model effectively addresses the "plausibility trap," where fluent but factually incorrect summaries mislead readers.
4. Conclusion and Future Work
This research demonstrates that a specialized post-editing model can effectively mitigate factual errors regarding speaker identity in dialogue summarization. By decoupling the generation of the gist from the verification of the participants, the system achieves higher factual reliability. Future work may expand this "detect-then-correct" paradigm to other factual dimensions, such as temporal data, locations, and logical negations, potentially creating a comprehensive fact-checking layer for abstractive summarization systems.
Summary updated