Improving Faithfulness in Abstractive Text Summarization with EDUs Using BART (Student Abstract)

Narjes Delpisheh, Yllias ChaliAAAI 2024 · DOI

Abstract

Abstractive text summarization models often struggle with factual inconsistency, or "hallucinations," where generated summaries deviate from the source material. This paper proposes a novel approach to improve faithfulness by utilizing Elementary Discourse Units (EDUs) as a granular guidance signal. By integrating EDUs into the GSum framework using a BART-based model, the researchers demonstrate significant improvements in both summary quality (ROUGE scores) and factual accuracy (Entailment and QAGS) over existing state-of-the-art methods on the CNN/Daily Mail dataset.

Introduction

While abstractive summarization offers the potential for fluent and concise document summaries, its practical utility is limited by the tendency of models to misinterpret relationships between ideas or invent facts. Previous attempts to mitigate this issue have used "highlighted sentences" as guidance. However, sentences are often too long and contain irrelevant "noise" that can confuse the model.

This research addresses the granularity problem by proposing the use of Elementary Discourse Units (EDUs)—the smallest semantically precise units of meaning, such as clauses or phrases. The authors hypothesize that EDUs provide a cleaner, more focused signal for the model to follow, leading to more faithful summaries.

Methodology

The proposed pipeline combines three core components:

  1. BART: A high-performing pre-trained language model serves as the generative engine.
  2. EDUs (via DISCOBERT): The source text is segmented into EDUs using DISCOBERT. These units represent complete, independent thoughts, allowing for better identification of inter-sentence relations like causality.
  3. GSum Framework: This guided summarization framework allows the BART model to accept an additional input signal (the EDUs) to steer the generation process.

During training, the model uses "Oracle" guidance derived from human-written reference summaries. At test time, the system employs MatchSum to identify important sentences and DISCOBERT to extract the corresponding crucial EDUs, which are then fed into the BART model.

Experimental Results

The researchers evaluated their approach against several strong baselines, including CLIFF (contrastive learning) and SEASON (salience-based guidance), using the CNN/Daily Mail benchmark.

Summary Quality

The BART+EDUs(G) model demonstrated superior performance in standard quality metrics. As shown in Table 1, it achieved a ROUGE-1 score of 0.54, significantly outperforming standard BART (0.44) and the SEASON baseline (0.46).

Table 1: Comparison of ROUGE scores across different summarization models.

Faithfulness and Semantic Similarity

To measure factual consistency, the authors utilized Entailment scores and QAGS (a question-answering consistency metric). The hybrid BART+Sents,EDUs(G) model, which combines sentence-level context with EDU-level precision, yielded the highest faithfulness scores in the study.

Table 2: Faithfulness and semantic similarity metrics for the proposed models and baselines.

As detailed in Table 2, the hybrid model achieved a 0.929 Entailment score and a 0.093 QAGS score. Furthermore, it showed superior semantic similarity to reference summaries with a SentSim score of 0.854. These results suggest that while EDUs provide the necessary precision for faithfulness, sentences may still provide valuable contextual information.

Discussion and Conclusion

The findings highlight that granularity is key to guiding abstractive models. By breaking text into EDUs, models can better preserve the semantic integrity of the source document.

However, some limitations remain. The computational overhead of dual-encoding and EDU extraction was not extensively discussed, and the study was limited to news data. Future work should explore the scalability of EDU-based guidance in more complex domains, such as legal or medical documentation, where factual precision is paramount. Overall, this approach offers a promising path toward more reliable and trustworthy AI-generated summaries.

Summary updated

BibTeX
@inproceedings{delpisheh2024improving,
  author = {Narjes Delpisheh and Yllias Chali},
  title = {{Improving Faithfulness in Abstractive Text Summarization with EDUs Using BART (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30433},
}
Figure