CAR-Transformer: Cross-Attention Reinforcement Transformer for Cross-Lingual Summarization

Yuang Cai, Yuyu YuanAAAI 2024 · DOI

TL;DR

This paper introduces CAR-Transformer, a novel approach to cross-lingual summarization (CLS) that uses a reinforcement learning-inspired mechanism to align cross-attention weights between source documents and their summaries. By leveraging the monolingual summary as a "pseudo-summary policy," CAR-Transformer achieves improved CLS performance without increasing model parameters or computational cost during inference.

What Problem Does This Paper Address?

Cross-lingual summarization (CLS) aims to generate a summary of a document in a different language. Existing methods often rely on complex multi-task training or knowledge distillation, which can be computationally expensive and lack interpretability. While these methods improve performance, they often struggle with error accumulation and high inference latency. A key challenge is effectively aligning the source document's content with the target language summary, especially when dealing with multiple languages. The paper addresses the need for a more efficient and explainable method to achieve this alignment.

Methods

The core innovation is the Cross-Attention Reinforcement (CAR) module. This module is integrated into a standard Transformer architecture. Instead of directly training for CLS, CAR formulates a "pseudo-summary policy" based on the cross-attention weights learned by the CLS model. This policy, parameterized by these weights, generates a pseudo-summary in the monolingual language.

The CAR module then uses a reinforcement learning approach to train this policy. The objective is to maximize a reward signal that measures the similarity between the generated pseudo-summary and the ground-truth monolingual summary. This process is called "cross-attention reinforcement" because it incentivizes the model to pay more attention to tokens in the source document that are relevant to the monolingual summary.

To stabilize the reinforcement learning process, the authors also incorporate a self-critical baseline. This baseline uses the reward from a greedily generated pseudo-summary to reduce variance in the gradient estimation. The final training objective combines the standard cross-entropy loss for CLS with this CAR loss. Crucially, the CAR module adds no extra parameters and is disabled during inference, meaning no additional computational cost.

Datasets

The paper evaluates CAR-Transformer on three benchmark datasets:

  • WikiLingua: This dataset provides parallel monolingual summaries for cross-lingual summarization.
  • GlobalVoice: This dataset contains cross-lingual summaries but lacks parallel monolingual summaries. The authors used an mT5 model fine-tuned on XLSum to generate these.
  • CrossSum: Similar to GlobalVoice, this dataset also lacks parallel monolingual summaries, and the authors used the same mT5-based generation method.

The WikiLingua dataset was used for experiments across multiple language pairs, including Korean-to-English, Hindi-to-English, Czech-to-English, Chinese-to-English, Thai-to-English, and Turkish-to-English. GlobalVoice and CrossSum were used for French-to-English and Arabic-to-English summarization.

Key Insights and Ideas

  • Reinforcement Learning for Attention Alignment: Applying policy gradient methods from RL to optimize cross-attention weights for CLS is a novel approach. It offers a way to "teach" the model where to focus without explicit supervision for every attention weight.
  • Pseudo-Summary Policy: The idea of using a pseudo-summary generated from attention weights as an intermediate step for reinforcement learning is a clever way to leverage the monolingual summary without needing a full MS model.
  • Explainability through Attention: The paper demonstrates that CAR-Transformer leads to more focused cross-attention, visually highlighting how the model attends to relevant parts of the source document for the summary. This addresses the interpretability gap in previous methods.
  • Efficiency without Extra Parameters: The CAR module's ability to improve performance without adding model parameters or increasing inference cost is a significant practical advantage.

Key Results

  • Improved Performance: CAR-Transformer consistently outperforms vanilla mBART fine-tuning and several multi-task training baselines (CLS-MS, CLS-MT, 2-step) across most CLS tasks.
  • WikiLingua: The approach achieved significant improvements, outperforming the best multi-task training approach by 1.48 in ROUGE-L F1 score on the Korean-to-English task.
  • Robustness: The paper shows consistent improvements across eight different languages, indicating the robustness of the CAR approach.
  • Interpretability: Visualizations show that CAR-Transformer's cross-attention is more focused on relevant source segments compared to CLS-MS. The Key Awareness Ratio (KAR) metric also shows CAR achieving the best score.
  • Training Cost: While CAR-Transformer's total training time is slightly higher due to reward computation (which runs on CPU), it avoids the increased computational cost of training separate MS or MT models.

Limitations and Future Work

  • CrossSum Performance: The paper notes that CAR-Transformer did not achieve significant improvements on the CrossSum dataset, though it still outperformed other approaches. This suggests potential dataset-specific challenges or limitations.
  • Training Time: The reward computation for the CAR module, which involves ROUGE scoring, cannot be parallelized on GPUs and thus adds to the overall training time compared to standard Transformer training.
  • Future Work: The authors suggest exploring more advanced reinforcement learning techniques and investigating the application of CAR to other cross-lingual sequence-to-sequence tasks. Further research into optimizing the reward computation for faster training could also be beneficial.

Summary updated

BibTeX
@inproceedings{cai2024car,
  author = {Yuang Cai and Yuyu Yuan},
  title = {{CAR-Transformer: Cross-Attention Reinforcement Transformer for Cross-Lingual Summarization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i16.29724},
}
Figure