ZmBART: An Unsupervised Cross-lingual Transfer Framework for Language Generation

Kaushal Kumar Maurya, Maunendra Sankar Desarkar, Yoshinobu Kano, Kumari Deepshikha

TL;DR

ZmBART is an unsupervised cross-lingual transfer framework for natural language generation (NLG) that eliminates the need for parallel corpora or back-translation. By introducing an innovative unsupervised auxiliary task and freezing specific model components during training, it successfully transfers task-specific supervision from English to low-resource languages like Hindi and Japanese in zero- and few-shot settings.

What Problem Does This Paper Address?

Natural Language Generation (NLG) tasks—such as text summarization, question generation, and headline generation—typically require massive amounts of task-specific annotated data. While these datasets are abundant for high-resource languages (HRLs) like English, they are scarce or non-existent for low-resource languages (LRLs) like Hindi and Japanese.

Existing cross-lingual transfer methods rely heavily on Machine Translation (MT) pipelines to create pseudo-parallel data or to translate inputs and outputs back and forth. However, MT systems for LRLs are often error-prone, do not share parameters across languages, and suffer from severe quality degradation. The authors tackle this gap by proposing a method to achieve direct cross-lingual supervision transfer without relying on any parallel text or translation steps.

Methods

The ZmBART framework introduces a multi-step pipeline built on top of a pre-trained multilingual model:

  1. Base Architecture: The framework leverages mBART, a massively multilingual sequence-to-sequence denoising autoencoder.
  2. Unsupervised Auxiliary Task ("rand-summary"): To bridge the gap between mBART's standard denoising pre-training objective and the auto-regressive nature of downstream NLG tasks, the authors introduce an intermediate pre-training step. Using only monolingual data from the target languages, the model is trained to take an input passage and generate a random selection of 20% of its sentences. This warms up the encoder-decoder weights for generation tasks.
  3. Fine-Tuning & Freezing: The model is fine-tuned on large, task-specific English datasets. To prevent the model from completely overwriting its multilingual capabilities (a phenomenon known as catastrophic forgetting), the authors freeze all word embeddings and decoder layers during this phase.
  4. Data Augmentation for Spurious Correlation: For tasks like abstractive summarization, zero-shot outputs often become "code-mixed" (a messy blend of English and the target language). The authors mitigate this by mixing a tiny amount (just 25 examples) of the unsupervised auxiliary task data into the English fine-tuning phase.
Figure 2: Architecture diagram of ZmBART

Datasets

The framework was evaluated across four distinct NLG tasks using three languages (English, Hindi, and Japanese):

  • News Headline Generation (NHG): Gigaword (English), Kaggle short summarization corpus (Hindi), and the Iwama & Kano 2019 dataset (Japanese).
  • Question Generation (QG): SQuAD 1.1 (English), MLQA & TyDiQA-GoldP (Hindi), and the Takahashi et al. 2019 dataset (Japanese).
  • Abstractive Text Summarization (ATS): The WikiLingua dataset for all three languages.
  • Distractor Generation (DG): Maurya & Desarkar 2020 dataset (English). Crucially, the authors introduce HiDG, a new high-quality distractor generation dataset for Hindi, created by translating and manually correcting SQuAD-based triplets.

Key Insights and Ideas

  • The Power of the Auxiliary Task: Simply fine-tuning mBART on English and testing it on Hindi or Japanese leads to poor lexical match scores because the decoder defaults to code-mixed text. The "rand-summary" auxiliary task acts as a vital bridge, teaching the decoder to generate coherent text in the correct target language using only monolingual data.
  • Freezing is Essential: Without freezing the decoder layers and word embeddings during English fine-tuning, the model completely succumbs to catastrophic forgetting, biasing the generator entirely toward English.
  • Few-Shot Efficiency: While zero-shot performance is impressive, adding as few as 1,000 supervised examples in the target language yields massive performance gains, with benefits plateauing quickly afterward.

Key Results

ZmBART consistently outperformed standard mBART baselines and machine translation pipelines across zero-shot tasks:

  • Hindi NHG: ZmBART achieved a ROUGE-L score of 32.74 in the zero-shot setting, outperforming the MT pipeline baseline by over 16 points.
Table 1: Zero and few-shot cross-lingual generation results for Hindi Language
  • Japanese NHG: ZmBART scored 19.92 ROUGE-L in zero-shot, jumping significantly to 31.55 in the few-shot setting.
Table 2: Zero and few-shot cross-lingual generation results for Japanese Language
  • Ablation Studies: Removing the auxiliary task caused ROUGE scores to plummet (e.g., Hindi ATS ROUGE-1 dropped from 21.27 to 4.34), proving its necessity for proper target-language alignment.
Ablation results showing the impact of the auxiliary task

Limitations and Future Work

Despite its success, ZmBART exhibited some language-specific quirks. In the Question Generation task for Japanese, the model frequently started questions with English "wh-" words, leading to lower manual evaluation scores from native speakers. This highlights that stark syntactic differences between the HRL and LRL can still pose challenges in pure zero-shot transfer.

For future work, the authors plan to extend the framework to include more diverse languages and explore alternative unsupervised auxiliary tasks to optimize cross-lingual knowledge transfer further.

Summary updated

Abstract

Despite the recent advancement in NLP research, cross-lingual transfer for natural language generation is relatively understudied. In this work, we transfer supervision from high resource language (HRL) to multiple low-resource languages (LRLs) for natural language generation (NLG). We consider four NLG tasks (text summarization, question generation, news headline generation, and distractor generation) and three syntactically diverse languages, i.e., English, Hindi, and Japanese. We propose an unsupervised cross-lingual language generation framework (called ZmBART) that does not use any parallel or pseudo-parallel/back-translated data. In this framework, we further pre-train mBART sequence-to-sequence denoising auto-encoder model with an auxiliary task using monolingual data of three languages. The objective function of the auxiliary task is close to the target tasks which enriches the multi-lingual latent representation of mBART and provides good initialization for target tasks. Then, this model is fine-tuned with task-specific supervised English data and directly evaluated with low-resource languages in the Zero-shot setting. To overcome catastrophic forgetting and spurious correlation issues, we applied freezing model component and data argumentation approaches respectively. This simple modeling approach gave us promising results.We experimented with few-shot training (with 1000 supervised data points) which boosted the model performance further. We performed several ablations and cross-lingual transferability analyses to demonstrate the robustness of ZmBART.
BibTeX
@inproceedings{maurya2021zmbart,
  author = {Kaushal Kumar Maurya and Maunendra Sankar Desarkar and Yoshinobu Kano and Kumari Deepshikha},
  title = {{ZmBART: An Unsupervised Cross-lingual Transfer Framework for Language Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}

Loading related content...

Figure