Abstract
Current state-of-the-art cross-lingual summarization models employ a multi-task learning paradigm, which works on a shared vocabulary module and relies on the self-attention mechanism to attend among tokens in two languages. However, correlation learned by self-attention is often loose and implicit, making it inefficient in capturing crucial cross-lingual representations. This challenge is exacerbated when dealing with languages with separate morphological or structural features, leading to performance drops. To overcome this, we propose a novel Knowledge-Distillation-based framework for Cross-Lingual Summarization (CLS). Our approach explicitly constructs cross-lingual correlation by distilling knowledge from a monolingual summarization teacher into a cross-lingual summarization student. We further propose a Knowledge Distillation loss using Sinkhorn Divergence, an Optimal-Transport distance, to estimate the discrepancy between teacher and student representations. Due to the geometric nature of Sinkhorn Divergence, the student model learns to align its produced cross-lingual hidden states with monolingual hidden states, leading to a strong correlation between distant languages. Experiments demonstrate that our method outperforms state-of-the-art models under both high and low-resourced settings.
Introduction
Cross-Lingual Summarization (CLS) is a complex task that requires a model to simultaneously understand a source document and generate a summary in a different target language. While existing models leverage multi-task learning, they often struggle to create a "tight" correlation between structurally distant languages, such as English and Chinese. In these cases, internal representations for each language often reside in disparate regions of the vector space, resulting in summaries that may lose the original topic or miss key information.
Methodology
To address these alignment issues, we developed a Knowledge Distillation (KD) framework consisting of a Monolingual Summarization (MLS) Teacher and a Cross-Lingual Summarization Student. The Teacher, trained on monolingual data (e.g., English to English), acts as an expert in the summarization task. The Student is then tasked with generating summaries across languages (e.g., English to Chinese).
The core innovation lies in the use of Sinkhorn Divergence as the distillation loss. Standard metrics like Euclidean distance or KL Divergence often fail when comparing data from different vector spaces. Sinkhorn Divergence, an Optimal Transport metric, treats hidden states as probability distributions and calculates the "work" required to transform one into the other. This geometrically aware approach forces the student model to align its cross-lingual hidden states with the teacher’s monolingual hidden states, effectively bridging the gap between different language spaces.
Experimental Setup
We evaluated our method on several challenging language pairs known for structural differences, including English-Chinese (En2Zh, Zh2En) and various Wikilingua pairs (English to Arabic, Japanese, and Vietnamese). To test the robustness of the model, we conducted experiments across various data availability scenarios, ranging from "Minimum" (as few as 1,500 samples) to "Full" dataset sizes.
Key Results
The proposed model consistently outperformed previous benchmarks, such as NCLS and MCLAS, across all metrics. On the Zh2En dataset, our model achieved significant improvements in ROUGE scores, demonstrating the effectiveness of explicit alignment via Sinkhorn Divergence.
Key findings include:
- Quantitative Gains: In the En2Ar (Arabic) task, the model saw a 5.1 point jump in ROUGE-L compared to previous state-of-the-art methods.
- Low-Resource Efficiency: Even with extremely limited training data (1,500 samples for En2Zh), the model outperformed baselines, proving that the teacher's knowledge effectively compensates for data scarcity.
- Human Evaluation: Blind tests indicated that human evaluators preferred our model's summaries for their superior informativeness and fluency.
Conclusion
By employing Sinkhorn Divergence within a Knowledge Distillation framework, we provide a robust method for aligning representations across distant languages in cross-lingual summarization. While the computational cost of Optimal Transport is higher than standard losses, the gains in summary quality and performance in low-resource settings are substantial. Future work may involve extending this framework to other cross-lingual tasks such as question answering and reasoning.
Summary updated