Abstract
The goal of the cross-lingual summarization (CLS) is to convert a document in one language to a summary in another one. Essentially, the CLS task is the combination of machine translation (MT) and monolingual summarization (MS), and thus there exists the hierarchical relationship between MT&MS and CLS. Existing studies on CLS mainly focus on utilizing pipeline methods or jointly training an end-to-end model through an auxiliary MT or MS objective. However, it is very challenging for the model to directly conduct CLS as it requires both the abilities to translate and summarize. To address this issue, we propose a hierarchical model for the CLS task, based on the conditional variational auto-encoder. The hierarchical model contains two kinds of latent variables at the local and global levels, respectively. At the local level, there are two latent variables, one for translation and the other for summarization. As for the global level, there is another latent variable for cross-lingual summarization conditioned on the two local-level variables. Experiments on two language directions (English-Chinese) verify the effectiveness and superiority of the proposed approach. In addition, we show that our model is able to generate better cross-lingual summaries than comparison models in the few-shot setting.
Introduction
Cross-lingual summarization (CLS) represents a significant challenge in Natural Language Processing (NLP), requiring a model to ingest a document in a source language (e.g., English) and produce a concise summary in a target language (e.g., Chinese). Traditionally, this has been approached via pipeline methods—translating then summarizing, or vice versa. However, these methods are prone to error propagation. While end-to-end models mitigate this, they often struggle to simultaneously master the distinct skills of translation and summarization. This paper introduces the Variational Hierarchical Model (VHM), which explicitly models the hierarchical relationship between these sub-tasks.
Methodology: The Variational Hierarchical Model
The VHM architecture is built upon the Transformer but incorporates a sophisticated Conditional Variational Auto-Encoder (CVAE) framework to manage the complexity of the CLS task.
The core innovation is its multi-level latent variable structure:
- Local Latent Variables: The model employs two separate latent variables, $z_{mt}$ for machine translation and $z_{ms}$ for monolingual summarization. These variables capture the specific data distributions of their respective tasks.
- Global Latent Variable: A third variable, $z_{cls}$, is introduced for the final cross-lingual task. Crucially, this global variable is conditioned on the two local variables, ensuring the CLS task is grounded in the fundamental abilities to translate and summarize.
- Training Objective: The model maximizes a variational lower bound (ELBO). Recognition networks guide prior networks to predict latent variables from the source text during inference, allowing the model to handle the inherent "one-to-many" nature of summarization.
Experimental Results
The VHM was evaluated on the Zh2EnSum (Chinese-to-English) and En2ZhSum (English-to-Chinese) benchmarks, utilizing auxiliary datasets like LCSTS and LDC for multi-task training.
Performance and Few-Shot Resilience
The VHM consistently outperformed state-of-the-art models such as ATS-A and MS-CLS. On the Zh2EnSum test set, the VHM achieved a ROUGE-1 score of 41.36. When initialized with pre-trained mBART weights, this improved to 43.97.
A key finding is the model's performance in data-scarce environments. As shown in the figures above, when CLS training data is reduced to as little as 0.1%, VHM maintains significantly higher performance than baselines. This resilience stems from its ability to leverage "sub-skills" learned from larger, independent MT and MS datasets.
Human Evaluation
In blind qualitative assessments, human graders consistently rated VHM summaries higher in Informativeness, Conciseness, and Fluency compared to baseline models in both full-data and few-shot settings.
Conclusion and Future Work
By treating cross-lingual summarization as a hierarchical problem, the VHM effectively bridges the gap between translation and summarization. While the model requires careful training (e.g., KL-annealing) to avoid posterior collapse, its success in few-shot learning marks a significant advancement for low-resource NLP tasks. Future research may extend this hierarchical variational approach to long-document or multi-document cross-lingual summarization.
Summary updated