ZmBART: An Unsupervised Cross-lingual Transfer Framework for Language Generation

Kaushal Kumar Maurya, Maunendra Sankar Desarkar, Yoshinobu Kano, Kumari Deepshikha

TL;DR

This paper introduces ZmBART, a novel unsupervised framework for cross-lingual natural language generation (NLG) that transfers knowledge from high-resource languages (like English) to low-resource languages (like Hindi and Japanese) without requiring any parallel data. ZmBART achieves this by pre-training a multilingual model (mBART) with a custom auxiliary task and then fine-tuning it on task-specific English data, enabling zero-shot generation in other languages.

What Problem Does This Paper Address?

Natural Language Generation (NLG) models typically require large amounts of annotated data, which are readily available for high-resource languages like English. However, for low-resource languages (LRLs), obtaining such data is challenging due to the time and cost involved in manual annotation. This disparity limits the development and deployment of advanced NLG applications for LRLs. While cross-lingual representation learning has advanced, cross-lingual transfer for generation tasks remains relatively underexplored, especially without relying on parallel or pseudo-parallel data. This paper aims to bridge this gap by enabling effective cross-lingual transfer for various NLG tasks to LRLs in zero-shot and few-shot settings.

Methods

The core of ZmBART is built upon the pre-trained multilingual BART (mBART) model. The authors introduce two key innovations:

  1. Unsupervised Auxiliary Task: To overcome the mismatch between mBART's pre-training objectives (denoising) and the auto-regressive nature of NLG tasks, they introduce an unsupervised auxiliary task. This task uses only monolingual data from the target languages and is designed to enrich the multilingual latent representations and train the decoder in an auto-regressive manner. Specifically, the auxiliary task involves generating a "rand-summary" by randomly selecting 20% of sentences from a given passage. This task is close in objective to the downstream NLG tasks and provides a better warm-start.
  2. Zero-Shot Transfer: After the auxiliary pre-training, the model is fine-tuned on task-specific supervised English data. Crucially, this fine-tuned model is then directly evaluated on Hindi and Japanese without any further training or parallel data for these languages (zero-shot setting).

To address catastrophic forgetting (where the model loses its pre-trained knowledge during fine-tuning) and spurious correlation issues, they employ two strategies: freezing model components (word embeddings and decoder layers) and using a small amount of auxiliary task data during fine-tuning.

Datasets

The paper utilizes several datasets:

  • English Datasets:
    • Gigaword for News Headline Generation (NHG).
  • SQuAD 1.1 for Question Generation (QG).
  • WikiLingua for Abstractive Text Summarization (ATS).
  • A custom English Distractor Generation (DG) dataset from Maurya and Desarkar (2020).
  • Hindi Datasets:
    • MLQA and TyDiQA-GoldP for QG.
  • A custom Hindi Distractor Generation dataset called HiDG (created by translating distractors from the English DG dataset and manually verifying them).
  • Kaggle dataset for NHG.
  • WikiLingua for ATS.
  • Japanese Datasets:
    • Takahashi et al. (2019) for QG.
  • Iwama and Kano (2019) for NHG.
  • WikiLingua for ATS.

The paper also introduces the HiDG dataset for distractor generation in Hindi, which is a novel contribution.

Key Insights and Ideas

  • Auxiliary Task is Crucial: The auxiliary task plays a vital role in enabling effective cross-lingual transfer. Without it, the model tends to generate code-mixed outputs or default to English, even when conditioned on other language tags.
  • Zero-Shot is Achievable Without Parallel Data: The framework demonstrates that high-quality cross-lingual generation is possible in a zero-shot setting without relying on any parallel or back-translated data, which are significant bottlenecks in previous approaches.
  • Catastrophic Forgetting Mitigation Works: Freezing model components and using auxiliary data augmentation effectively combats catastrophic forgetting, allowing the model to retain its cross-lingual capabilities.
  • Robustness Across Tasks and Languages: ZmBART shows consistent performance improvements across diverse NLG tasks (NHG, QG, ATS, DG) and typologically different languages (English, Hindi, Japanese), highlighting its generalizability.
  • Few-Shot Boosts Performance: While zero-shot performance is promising, fine-tuning with a small amount of task-specific data (even just 1000 examples) significantly boosts performance, especially for LRLs.

Key Results

  • Hindi NHG: ZmBART outperforms baselines with an absolute difference of 5.22 ROUGE-L score in zero-shot settings.
  • Cross-lingual Transfer: ZmBART consistently outperforms MT pipeline baselines across various tasks and languages in zero-shot settings, even without parallel data.
  • Few-Shot Improvement: Few-shot training (with 1000 data points) significantly boosts performance, with Hindi language few-shot results for ATS reaching 27.22 ROUGE-L and for QG reaching 33.49 BERTScore.
  • Manual Evaluation: Human evaluations generally align with automated metrics, showing ZmBART's effectiveness, particularly for Hindi outputs. Japanese outputs sometimes received lower scores, attributed to code-mixing or stylistic differences.
  • Auxiliary Task Impact: Table 6 shows that ZmBART with the auxiliary task significantly outperforms the version without it for both Hindi and Japanese in zero-shot settings for ATS and QG.

Limitations and Future Work

  • Code-Mixing in Japanese: While ZmBART's outputs for Hindi were often logical and meaningful even when code-mixed, Japanese outputs sometimes showed a higher degree of English code-mixing, which was less accepted by annotators.
  • Task-Specific Nuances: The paper notes that while the auxiliary task is general, some NLG tasks (like ATS and NHG) might benefit more from abstractive capabilities, while others (like QG and DG) have objectives that are more distinct from the auxiliary task.
  • Future Work: The authors plan to extend this work by:
  • Adding support for multiple other languages and NLG tasks.
  • Exploring other choices of auxiliary tasks for even better model transfer.
  • Investigating more sophisticated approaches to mitigate catastrophic forgetting and spurious correlations.

Summary updated

Abstract

Despite the recent advancement in NLP research, cross-lingual transfer for natural language generation is relatively understudied. In this work, we transfer supervision from high resource language (HRL) to multiple low-resource languages (LRLs) for natural language generation (NLG). We consider four NLG tasks (text summarization, question generation, news headline generation, and distractor generation) and three syntactically diverse languages, i.e., English, Hindi, and Japanese. We propose an unsupervised cross-lingual language generation framework (called ZmBART) that does not use any parallel or pseudo-parallel/back-translated data. In this framework, we further pre-train mBART sequence-to-sequence denoising auto-encoder model with an auxiliary task using monolingual data of three languages. The objective function of the auxiliary task is close to the target tasks which enriches the multi-lingual latent representation of mBART and provides good initialization for target tasks. Then, this model is fine-tuned with task-specific supervised English data and directly evaluated with low-resource languages in the Zero-shot setting. To overcome catastrophic forgetting and spurious correlation issues, we applied freezing model component and data argumentation approaches respectively. This simple modeling approach gave us promising results.We experimented with few-shot training (with 1000 supervised data points) which boosted the model performance further. We performed several ablations and cross-lingual transferability analyses to demonstrate the robustness of ZmBART.
BibTeX
@inproceedings{maurya2021zmbart,
  author = {Kaushal Kumar Maurya and Maunendra Sankar Desarkar and Yoshinobu Kano and Kumari Deepshikha},
  title = {{ZmBART: An Unsupervised Cross-lingual Transfer Framework for Language Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure