A Variational Hierarchical Model for Neural Cross-Lingual Summarization

Yunlong Liang, Fandong Meng, Chulun Zhou, Jinan Xu, Yufeng Chen, Jinsong Su, Jie Zhou

Abstract

Cross-lingual summarization (CLS) is a challenging task that requires both information condensation and language translation. We propose the Variational Hierarchical Model (VHM), which treats CLS as a structured synthesis of its two core components: machine translation (MT) and monolingual summarization (MS). By utilizing a hierarchical Conditional Variational Auto-Encoder (CVAE), the model learns specialized latent variables for translation and summarization, which then inform a global variable for the final task. This approach leads to state-of-the-art performance, particularly in data-scarce, few-shot scenarios.

Introduction

Cross-lingual summarization (CLS) presents a unique challenge in natural language processing. To succeed, a model must simultaneously condense information (summarization) and convert it into another language (translation). Historically, researchers have relied on two main approaches: sequential pipelines (translate then summarize, or vice versa) and end-to-end multi-tasking. Pipelines are often plagued by error propagation, while standard multi-tasking models frequently fail to capture the explicit hierarchical relationship between the auxiliary tasks and the final goal.

The authors argue that translation and summarization should be treated as the fundamental building blocks of a cross-lingual summary rather than mere auxiliary tasks. This insight led to the development of VHM, which explicitly models these dependencies to improve performance when labeled CLS data is limited.

The Variational Hierarchical Model (VHM)

The core innovation of this work is the hierarchical structure of the latent space within a CVAE framework. The architecture is designed to force the model to learn how translation and summarization features combine to form a cross-lingual summary.

Architecture of the VHM model showing the hierarchical relationship between local and global latent variables.

Architecture Components:

  • Encoder: A standard Transformer encoder processes the source document.
  • Local Latent Variables: The model introduces two "local" latent variables: $z_{mt}$ (specialized for translation) and $z_{ms}$ (specialized for summarization). During training, these variables are constrained to reconstruct the actual translation and the source-language summary, respectively.
  • Global Latent Variable: A "global" variable, $z_{cls}$, is generated by conditioning on both the input text and the two local variables ($z_{mt}$ and $z_{ms}$).
  • Decoder: The decoder utilizes the global latent variable to produce the final target-language summary.

To prevent "posterior collapse"—a common issue in VAEs where the model ignores latent variables—the authors employ KL annealing, gradually increasing the weight of the regularization term during the training process.

Experimental Evaluation

The VHM was evaluated on two major benchmarks: Zh2EnSum (Chinese-to-English, ~1.7M pairs) and En2ZhSum (English-to-Chinese, ~370k pairs). To support the training of local latent variables, auxiliary datasets for Chinese and English summarization (LCSTS, ENSUM) and machine translation (LDC corpora) were utilized.

Results and Analysis

The VHM consistently outperformed strong baselines, including the previous state-of-the-art model, ATS-A. When initialized with mBART, the VHM achieved the highest ROUGE scores reported on these datasets to date. Human evaluations further confirmed these findings, with graders rating VHM summaries as more informative, concise, and fluent.

Few-Shot Dominance

One of the most significant findings is the model's robustness in data-scarce scenarios. Because the VHM learns the underlying mechanics of translation and summarization through its local variables, it can generate high-quality summaries even with minimal CLS-specific training data.

Performance of VHM vs. baselines across different proportions of training data on the Zh2EnSum dataset.

As shown in the charts above, when training data is reduced to as little as 0.1% (approximately 1,700 examples), the performance gap between VHM and standard multi-task models widens significantly. This demonstrates that the hierarchical structure effectively transfers knowledge from high-resource auxiliary tasks to the lower-resource CLS task.

Conclusion

The VHM demonstrates that explicitly modeling the hierarchical dependency between translation, summarization, and cross-lingual summarization is superior to parallel multi-tasking. While the model introduces some computational complexity due to the multiple latent variables, its efficiency in few-shot learning makes it a powerful tool for expanding CLS to new language pairs. Future work may explore the model's performance on linguistically distant or extremely low-resource language pairs where even auxiliary data is scarce.

Summary updated

BibTeX
@inproceedings{liang2022variational,
  author = {Yunlong Liang and Fandong Meng and Chulun Zhou and Jinan Xu and Yufeng Chen and Jinsong Su and Jie Zhou},
  title = {{A Variational Hierarchical Model for Neural Cross-Lingual Summarization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.148},
}
Figure