Dialogue Inspectional Summarization with Factual Inconsistency Awareness

Leilei Gan, Yating Zhang, Kun Kuang, Lin Yuan, Shuo Li, Changlong Sun, Xiaozhong Liu, Fei WuarXiv 2021 · arXiv

Abstractive summarization has seen remarkable progress, yet "hallucinations"—the generation of factually incorrect information—remain a critical barrier to deployment in high-stakes domains. In legal or medical contexts, a minor factual error can have severe consequences. Multi-party dialogues present a unique challenge due to their non-linear nature, where key facts are often dispersed across multiple turns and speakers. A new framework, Dialogue Inspectional Summarization (DIS), addresses this by treating summarization as an inspectional task, ensuring that generated summaries are both factually grounded and logically complete.

The DIS Framework

To address the complexities of multi-party interactions, the researchers developed the DIS framework. This approach moves beyond simple sequence-to-sequence modeling by incorporating auxiliary tasks that force the model to respect factual constraints.

The architecture of the DIS framework, featuring the Multi-role Dialogue Encoder, EFAR, and MFED modules.

The framework consists of three primary components:

  • Multi-role Dialogue Encoder: This module uses a hierarchical structure to capture speaker-specific information, helping the model distinguish between the roles of Judge, Plaintiff, and Defendant. This role-awareness is crucial for understanding the intent and weight of different utterances.
  • Expectant Factual Aspect Regularization (EFAR): EFAR is a "look-ahead" task that predicts which factual categories (e.g., "Loan Amount," "Repayment Period") should appear in the summary based on the input dialogue. It acts as a regularizer, guiding the decoder to focus on essential factual aspects.
  • Missing Factual Entity Discrimination (MFED): This module acts as an alarm, identifying whether specific factual entities mentioned in a reference summary are actually present in the source dialogue. This helps the model recognize when it might be hallucinating information not supported by the transcript.

The Private Loan Disputes (PLD) Dataset

The authors benchmarked their framework on a newly curated dataset derived from over 300,000 real-world civil trial records. Legal experts labeled 45,531 cases, identifying 12 specific factual aspects and 14 types of factual entities. The task requires the model to generate a formal "fact-finding" paragraph from a noisy, colloquial trial transcript, making it a rigorous test for factual consistency.

Experimental Results

The DIS framework was integrated into several base models, including PEGASUS. The results show consistent improvements across standard automated metrics and human evaluations.

Performance comparison of various summarization methods on the PLD dataset using ROUGE and BERTScore.

As shown in the results table, PEGASUS-DIS achieves the highest scores in ROUGE-1 (55.64) and BERTScore F1 (86.73). Beyond automated metrics, human evaluations by legal experts confirmed that the DIS framework significantly enhances factual consistency (scoring 3.93/5.0 vs. 3.84/5.0 for the best baseline). An ablation study further highlighted that the EFAR module is particularly effective; removing it led to a notable decrease in performance, proving the value of forcing the model to "expect" certain factual categories.

Conclusion and Future Work

The DIS framework represents a significant step toward more reliable dialogue summarization in professional domains. By explicitly modeling factual expectations and identifying missing information, it provides a robust mechanism for reducing hallucinations. While the model still faces challenges with complex numerical reasoning and distinguishing between subtle factual conflicts in long dialogues, the integration of inspectional tasks offers a promising path forward for high-stakes NLP applications.

Summary updated

Abstract

Dialogue summarization has been extensively studied and applied, where the prior works mainly focused on exploring superior model structures to align the input dialogue and the output summary. However, for professional dialogues (e.g., legal debate and medical diagnosis), semantic/statistical alignment can hardly fill the logical/factual gap between input dialogue discourse and summary output with external knowledge. In this paper, we mainly investigate the factual inconsistency problem for Dialogue Inspectional Summarization (DIS) under non-pretraining and pretraining settings. An innovative end-to-end dialogue summary generation framework is proposed with two auxiliary tasks: Expectant Factual Aspect Regularization (EFAR) and Missing Factual Entity Discrimination (MFED). Comprehensive experiments demonstrate that the proposed model can generate a more readable summary with accurate coverage of factual aspects as well as informing the user with potential missing facts detected from the input dialogue for further human intervention.
BibTeX
@misc{gan2021dialogue,
  author = {Leilei Gan and Yating Zhang and Kun Kuang and Lin Yuan and Shuo Li and Changlong Sun and Xiaozhong Liu and Fei Wu},
  title = {{Dialogue Inspectional Summarization with Factual Inconsistency Awareness}},
  year = {2021},
  eprint = {2111.03284},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.03284},
}
Figure