Cross-Lingual Summarization with Pseudo-Label Regularization

Thang LeNAACL 2024 · DOI

TL;DR

This paper tackles the challenge of creating accurate summaries in one language from documents written in another, a task known as Cross-Lingual Summarization (XLS). The authors propose a novel method using "pseudo-labels" (generated summaries from other models) to regularize training, significantly improving summary quality compared to standard methods that rely solely on a single "gold" reference summary.

What Problem Does This Paper Address?

The world is awash in information, and the ability to quickly grasp key points is crucial. Cross-Lingual Summarization (XLS) aims to break down language barriers by generating summaries in a target language from documents in a source language. However, existing XLS models often struggle because they are trained on only one "gold" summary per document. This limited exposure can lead to models that are too rigid and prone to errors, failing to capture the full spectrum of possible good summaries. The authors argue that this "one-point" hypothesis space is insufficient for robust XLS.

Methods

The core contribution of this paper is the introduction of pseudo-label regularization for XLS. Here's how it works:

  • Teacher Models: They use existing, well-performing summarization models (teacher models) to generate multiple candidate summaries (pseudo-references) for a given document.
  • Pseudo-Labels: These pseudo-references are then used to create "pseudo-labels," which are essentially probability distributions over the target summary tokens, reflecting the teacher model's confidence.
  • Regularization Loss: A new loss term, the KL-divergence between the base model's prediction and the pseudo-label, is added to the standard training objective. This encourages the base model to learn from a richer set of potential summaries, not just the single gold reference.
  • Ensemble and Dropout: To improve the quality and diversity of pseudo-labels, they explore using ensembles of teacher models and applying dropout to teacher models to generate stochastic labels.

Datasets

The primary dataset used in this study is WIKILINGUA, a large-scale dataset for cross-lingual summarization. They conduct experiments across 8 typologically diverse languages: English (EN), Vietnamese (VI), Japanese (JA), Chinese (ZH), Arabic (AR), Korean (KO), Russian (RU), and Turkish (TR).

Key Insights and Ideas

  • Pseudo-labels are powerful: The study strongly suggests that leveraging pseudo-labels from teacher models is a simple yet highly effective way to boost XLS performance.
  • Monolingual teachers are best: Surprisingly, monolingual summarizers often produce higher-quality pseudo-labels than cross-lingual ones, likely because they can exploit simpler shortcuts without the added complexity of cross-language alignment.
  • Diversity matters: Not only the quality but also the diversity of pseudo-references is beneficial for regularization.
  • More pseudo-references are better: Increasing the size of the pseudo-reference pool generally leads to improved results.
  • Soft labels are useful: Even when applied to gold summaries, soft probability labels (instead of one-hot) yield improvements, further validating the utility of probabilistic outputs.

Key Results

  • The proposed pseudo-label regularization consistently improves XLS performance across 8 diverse languages, outperforming standard gold-reference training.
  • In fully supervised settings, their method (GOLD+PSEUDO) achieves significant improvements, even surpassing models trained with much larger amounts of parallel data. For instance, they report a substantial rise of 2.2 points in ROUGE-L compared to the base GOLD model.
  • Human evaluations also favored summaries generated by their method, indicating better informativeness and fluency.
  • The effectiveness of pseudo-labeling is demonstrated across different model architectures (MBART-50, Transformer, PISCES) and in few-shot and parameter-efficient fine-tuning scenarios.

Limitations and Future Work

The authors acknowledge several limitations:

  • Data dependency: Pseudo-labeling relies on the availability of parallel data or pre-trained models. In truly low-resource scenarios where such resources are scarce, its direct application might be challenging.
  • Domain generalization: Experiments were conducted on the WIKILINGUA dataset. Further research is needed to confirm if these findings generalize to other domains.
  • Evaluation aspects: While informativeness and fluency were assessed, other aspects like attribution and hallucination were not explicitly quantified.

Future work could explore constructing pseudo-parallel data for low-resource settings and investigating the impact of pseudo-labels on other summarization evaluation metrics.

Summary updated

BibTeX
@inproceedings{le2024cross,
  author = {Thang Le},
  title = {{Cross-Lingual Summarization with Pseudo-Label Regularization}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.289},
}
Figure