Abstractive summarization has seen remarkable progress, yet "hallucinations"—the generation of factually incorrect information—remain a critical barrier to deployment in high-stakes domains. In legal or medical contexts, a minor factual error can have severe consequences. Multi-party dialogues present a unique challenge due to their non-linear nature, where key facts are often dispersed across multiple turns and speakers. A new framework, Dialogue Inspectional Summarization (DIS), addresses this by treating summarization as an inspectional task, ensuring that generated summaries are both factually grounded and logically complete.
The DIS Framework
To address the complexities of multi-party interactions, the researchers developed the DIS framework. This approach moves beyond simple sequence-to-sequence modeling by incorporating auxiliary tasks that force the model to respect factual constraints.
The framework consists of three primary components:
- Multi-role Dialogue Encoder: This module uses a hierarchical structure to capture speaker-specific information, helping the model distinguish between the roles of Judge, Plaintiff, and Defendant. This role-awareness is crucial for understanding the intent and weight of different utterances.
- Expectant Factual Aspect Regularization (EFAR): EFAR is a "look-ahead" task that predicts which factual categories (e.g., "Loan Amount," "Repayment Period") should appear in the summary based on the input dialogue. It acts as a regularizer, guiding the decoder to focus on essential factual aspects.
- Missing Factual Entity Discrimination (MFED): This module acts as an alarm, identifying whether specific factual entities mentioned in a reference summary are actually present in the source dialogue. This helps the model recognize when it might be hallucinating information not supported by the transcript.
The Private Loan Disputes (PLD) Dataset
The authors benchmarked their framework on a newly curated dataset derived from over 300,000 real-world civil trial records. Legal experts labeled 45,531 cases, identifying 12 specific factual aspects and 14 types of factual entities. The task requires the model to generate a formal "fact-finding" paragraph from a noisy, colloquial trial transcript, making it a rigorous test for factual consistency.
Experimental Results
The DIS framework was integrated into several base models, including PEGASUS. The results show consistent improvements across standard automated metrics and human evaluations.
As shown in the results table, PEGASUS-DIS achieves the highest scores in ROUGE-1 (55.64) and BERTScore F1 (86.73). Beyond automated metrics, human evaluations by legal experts confirmed that the DIS framework significantly enhances factual consistency (scoring 3.93/5.0 vs. 3.84/5.0 for the best baseline). An ablation study further highlighted that the EFAR module is particularly effective; removing it led to a notable decrease in performance, proving the value of forcing the model to "expect" certain factual categories.
Conclusion and Future Work
The DIS framework represents a significant step toward more reliable dialogue summarization in professional domains. By explicitly modeling factual expectations and identifying missing information, it provides a robust mechanism for reducing hallucinations. While the model still faces challenges with complex numerical reasoning and distinguishing between subtle factual conflicts in long dialogues, the integration of inspectional tasks offers a promising path forward for high-stakes NLP applications.
Summary updated