Leveraging Entailment Judgements in Cross-Lingual Summarisation

Huajian Zhang, Laura Perez-BeltrachiniACL 2024 · DOI

Abstract

Cross-lingual summarization (CLS) faces a significant challenge: synthetic training data often contains "hallucinations" where reference summaries include information not present in the source document. This paper proposes a framework to mitigate this by using Cross-lingual Natural Language Inference (X-NLI) to detect unfaithful content. By training models with an unlikelihood loss (Unlike$_{PR}$), the authors demonstrate substantial improvements in summary faithfulness across diverse language pairs, including a newly introduced Chinese-English benchmark, while maintaining high levels of informativeness.

Introduction

The goal of Cross-Lingual Summarization is to generate a summary in a target language (e.g., English) from a source document in a different language (e.g., Czech). Due to the scarcity of human-annotated data, researchers typically rely on synthetic datasets aligned from multilingual sources like Wikipedia. However, these "gold" summaries are often noisy, containing facts absent from the source text. Models trained on such data learn to hallucinate, and standard metrics like ROUGE fail to penalize these factual inconsistencies.

Methodology

The authors introduce a two-stage pipeline focusing on Automatic Faithfulness Estimation and Faithfulness-Aware Training.

Automatic Faithfulness Estimation

To identify hallucinations, the researchers employ a multilingual Natural Language Inference (NLI) model (mT5-large). The model evaluates whether a summary sentence is logically entailed by the source document. The study utilizes INFUSE, a method that builds a supportive premise by incrementally selecting relevant document sentences. As shown in benchmarking results, INFUSE consistently outperforms other NLI-based estimators across various language pairs.

Comparison of NLI-based faithfulness estimation methods across language pairs.

Faithfulness-Aware Training

Three main strategies were compared to handle noisy data:

  1. Clean: Removing the most unfaithful document-summary pairs from the training set.
  2. Mask: Setting the loss to zero for tokens labeled as unfaithful, effectively ignoring them during backpropagation.
  3. Unlike$_{PR}$ (Proposed): This novel approach uses unlikelihood loss to explicitly penalize the generation of unfaithful sequences. It incorporates a "faithfulness prompt" (segment embeddings) to help the model distinguish between factual and hallucinated signals during training.

Datasets and Benchmarks

The study primarily uses the XWikis corpus (English, French, German, Czech). A significant contribution is the extension to XWikis-ZH, adding Chinese to provide a benchmark for a linguistically distant, lower-resource pair. For out-of-distribution testing, the Voxeurop news dataset was used. Additionally, the authors curated Test$_{Faith}$, a high-quality subset of the test data containing only faithful references for more rigorous evaluation.

Experimental Results

The proposed UnlikePR method demonstrated superior performance across all metrics and language pairs.

Automatic Evaluation

On the XWikis test sets, UnlikePR consistently achieved high faithfulness scores (measured by UniEval and INFUSE). Importantly, these gains did not compromise informativeness; ROUGE-L (RL) and LaBSE scores remained competitive with or superior to the vanilla baseline. The "Clean" variant also showed that removing the bottom 20% of noisy data improved performance, suggesting that data fidelity is often more important than volume.

Performance of various training strategies on the XWikis test sets.

Human Evaluation and Zero-Shot Transfer

Bilingual human judges rated UnlikePR significantly higher for faithfulness (e.g., 3.10 vs 2.13 for Vanilla in fr-en). Furthermore, the model's performance on the Voxeurop dataset suggests that the faithfulness-aware training generalizes well to new domains.

Key Insights

  • Robustness of Cross-Lingual NLI: Off-the-shelf NLI models proved surprisingly effective in zero-shot cross-lingual settings, often matching monolingual performance.
  • Effectiveness of Active Penalization: Actively teaching the model what not to generate via unlikelihood loss proved more powerful than simply filtering or masking bad data.
  • Domain Specificity: Hallucinations were more prevalent in lower-resource pairs like zh-en, likely due to differences in how Wikipedia articles are expanded across languages.

Conclusion and Future Work

This work highlights the importance of addressing data noise in CLS. While the current study used mBART50, the NLI-based filtering techniques could potentially be applied to Large Language Models (LLMs) for selecting high-quality in-context examples. Future research could explore more dynamic, sample-specific thresholding for identifying unfaithful content and further refine NLI models to reduce error propagation.

Summary updated

BibTeX
@inproceedings{zhang2024leveraging,
  author = {Huajian Zhang and Laura Perez-Beltrachini},
  title = {{Leveraging Entailment Judgements in Cross-Lingual Summarisation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.862},
}
Figure