FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction

Alessandro Scirè, Karim Ghonim, Roberto NavigliACL 2024 · DOI

Introduction

The rapid advancement of Large Language Models (LLMs) has enabled the generation of highly fluent and coherent summaries. However, these models frequently produce "hallucinations"—information that is plausible-sounding but factually inconsistent with the source text. Detecting these errors is critical for the deployment of summarization systems in high-stakes domains.

Existing evaluation metrics often struggle with interpretability, failing to pinpoint specific errors. Furthermore, they frequently exhibit a "short-form bias," performing poorly on longer documents where evidence is dispersed. Finally, the high computational cost of using large-scale LLMs as evaluators limits their practical application. To address these challenges, we present FENICE, a high-performance, interpretable, and efficient metric for factuality evaluation.

The FENICE Framework

FENICE (Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction) utilizes a modular pipeline to assess the truthfulness of generated summaries.

Overview of the FENICE evaluation pipeline.

The framework operates in three distinct stages:

  1. Claim Extraction: Rather than evaluating entire sentences, which may contain multiple facts, FENICE decomposes the summary into "Atomic Content Units" (ACUs). This is achieved using an LLM (GPT-3.5) or a distilled T5-base model, ensuring each claim is a short, independent statement.
  2. NLI-based Alignment: Each extracted claim is verified against the source document using a DeBERTa-v3-large Natural Language Inference (NLI) model. FENICE checks for entailment or contradiction at multiple levels of granularity—sentence, paragraph, and document. This multi-level approach is crucial for identifying facts synthesized from disparate parts of the text.
  3. Coreference Resolution: To prevent the NLI model from being misled by ambiguous pronouns, FENICE incorporates a coreference resolution step using FastCoref. By aligning entities (e.g., resolving "he" to "Billy Vunipola"), the metric significantly improves the precision of fact verification.

To optimize for production environments, the authors distilled the claim extraction capabilities of GPT-3.5 into a T5-base model (700M parameters). This distilled version maintains high accuracy while being significantly faster and more cost-effective.

Experimental Setup

The performance of FENICE was validated across several benchmarks:

  • AggreFact: A comprehensive benchmark combining multiple news-based datasets (CNN/DailyMail, XSum) with human factuality annotations.
  • ROSE: Utilized specifically to assess the quality of the claim extraction component.
  • Long-form Summarization: A newly curated dataset consisting of human annotations for 338 summaries of English narrative books, designed to test the metric's robustness on lengthy, complex documents.

Key Results

FENICE demonstrates superior performance across standard and long-form benchmarks. On the AggreFact benchmark, the GPT-based version of FENICE achieved a state-of-the-art (SOTA) balanced accuracy of 72.7%, outperforming established metrics such as AlignScore and SummaC.

Comparison of FENICE with state-of-the-art metrics on the AggreFact benchmark.

Notably, the distilled T5 version (FENICE_T5_claims) remained highly competitive, achieving an average score of 70.0%. This confirms that high-quality factuality checking can be achieved without relying on massive, expensive APIs.

A critical insight from the study is the importance of granularity. By allowing the NLI model to evaluate claims against paragraphs rather than just single sentences, the authors observed a performance increase of over 10 points. Furthermore, FENICE's atomic approach provides high interpretability, allowing users to visualize a "heat map" of a summary to identify specific hallucinations.

Conclusion

FENICE addresses the core limitations of current factuality metrics by combining atomic claim extraction with multi-level NLI alignment and coreference resolution. Its ability to handle long-form content and its efficiency through model distillation make it a robust tool for both research and real-world production systems. Future work will focus on expanding language support and further refining the alignment process through specialized fine-tuning.

Summary updated

BibTeX
@inproceedings{scire2024fenice,
  author = {Alessandro Scirè and Karim Ghonim and Roberto Navigli},
  title = {{FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.841},
}
Figure