Improving Neural Cross-Lingual Summarization via Employing Optimal Transport Distance for Knowledge Distillation

Thong Nguyen, Luu Anh Tuan

Abstract

Current state-of-the-art cross-lingual summarization models employ multi-task learning paradigm, which works on a shared vocabulary module and relies on the self-attention mechanism to attend among tokens in two languages. However, correlation learned by self-attention is often loose and implicit, inefficient in capturing crucial cross-lingual representations between languages. The matter worsens when performing on languages with separate morphological or structural features, making the cross-lingual alignment more challenging, resulting in the performance drop. To overcome this problem, we propose a novel Knowledge-Distillation-based framework for Cross-Lingual Summarization, seeking to explicitly construct cross-lingual correlation by distilling the knowledge of the monolingual summarization teacher into the cross-lingual summarization student. Since the representations of the teacher and the student lie on two different vector spaces, we further propose a Knowledge Distillation loss using Sinkhorn Divergence, an Optimal-Transport distance, to estimate the discrepancy between those teacher and student representations. Due to the intuitively geometric nature of Sinkhorn Divergence, the student model can productively learn to align its produced cross-lingual hidden states with monolingual hidden states, hence leading to a strong correlation between distant languages. Experiments on cross-lingual summarization datasets in pairs of distant languages demonstrate that our method outperforms state-of-the-art models under both high and low-resourced settings.

Introduction

Cross-Lingual Summarization (CLS) is the challenging task of understanding a document in a source language (e.g., Chinese) and generating a concise summary in a different target language (e.g., English). While current models utilize multi-task learning and self-attention to bridge the linguistic gap, these alignments are often implicit and "loose." This structural misalignment is particularly pronounced when dealing with distant language pairs or low-resource scenarios where training data is limited. This paper addresses these limitations by introducing a framework that explicitly aligns cross-lingual representations through geometric knowledge distillation.

Methods

The proposed framework leverages a Teacher-Student architecture to transfer summarization logic across languages. A "Teacher" model is first trained on a monolingual summarization task (e.g., Chinese-to-Chinese). Subsequently, a "Student" model is trained for the cross-lingual task (e.g., Chinese-to-English), aiming to mimic the internal logic of the teacher.

Architecture of the proposed Knowledge Distillation framework using Sinkhorn Divergence.

The core innovation lies in the use of Sinkhorn Divergence to align the hidden states of the teacher and student. Because these states exist in different vector spaces, standard distance metrics are insufficient. By treating hidden states as probability distributions and applying Sinkhorn Divergence—a computationally efficient version of Optimal Transport—the model calculates the cost of transforming student representations to match the teacher's. This geometric approach allows the student to learn the structural relationships between tokens across languages, making the model more robust to noise and outliers compared to traditional metrics like KL-Divergence.

Experimental Setup and Results

The model was evaluated on several benchmarks involving distant language pairs, including English-to-Chinese (En2Zh), Chinese-to-English (Zh2En), and subsets of Wikilingua (En2Ar, En2Ja, Ja2En, En2Vi). To test robustness, the researchers evaluated performance across "Minimum," "Medium," and "Maximum" data subsets.

Comparison of ROUGE scores against state-of-the-art baselines.

As shown in the results, the proposed model consistently outperforms previous state-of-the-art methods such as NCLS and MCLAS. On the Zh2En task, the model achieved a significant improvement in ROUGE-2 and ROUGE-L scores. In low-resource settings, the benefits were even more pronounced; for example, in the "Minimum" data setting for En2Zh, the model saw a jump of 3.6 ROUGE-1 points. Human evaluations further confirmed these findings, with participants preferring the proposed model's summaries for their superior informativeness and faithfulness to the source text.

Conclusion

By employing Optimal Transport for knowledge distillation, this framework effectively bridges the gap between distant languages in cross-lingual summarization. While the student's performance is currently bounded by the quality of the monolingual teacher, the geometric alignment approach proves highly effective, especially when data is scarce. Future work may extend this methodology to other cross-lingual tasks such as machine translation and cross-lingual retrieval.

Summary updated

Abstract

Current state-of-the-art cross-lingual summarization models employ multi-task learning paradigm, which works on a shared vocabulary module and relies on the self-attention mechanism to attend among tokens in two languages. However, correlation learned by self-attention is often loose and implicit, inefficient in capturing crucial cross-lingual representations between languages. The matter worsens when performing on languages with separate morphological or structural features, making the cross-lingual alignment more challenging, resulting in the performance drop. To overcome this problem, we propose a novel Knowledge-Distillation-based framework for Cross-Lingual Summarization, seeking to explicitly construct cross-lingual correlation by distilling the knowledge of the monolingual summarization teacher into the cross-lingual summarization student. Since the representations of the teacher and the student lie on two different vector spaces, we further propose a Knowledge Distillation loss using Sinkhorn Divergence, an Optimal-Transport distance, to estimate the discrepancy between those teacher and student representations. Due to the intuitively geometric nature of Sinkhorn Divergence, the student model can productively learn to align its produced cross-lingual hidden states with monolingual hidden states, hence leading to a strong correlation between distant languages. Experiments on cross-lingual summarization datasets in pairs of distant languages demonstrate that our method outperforms state-of-the-art models under both high and low-resourced settings.
BibTeX
@inproceedings{nguyen2022improving,
  author = {Thong Nguyen and Luu Anh Tuan},
  title = {{Improving Neural Cross-Lingual Summarization via Employing Optimal Transport Distance for Knowledge Distillation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=yiG86KR24k},
}
Figure