Cross-Lingual Training for Automatic Question Generation

Vishwajeet Kumar, Nitish Joshi, Arijit Mukherjee, Ganesh Ramakrishnan, Preethi Jyothi

TL;DR

This paper tackles the challenge of automatic question generation (QG) in low-resource languages by proposing a cross-lingual training approach. They leverage large English QG datasets to improve QG performance in languages like Hindi and Chinese, where such data is scarce, by using unsupervised pretraining and joint supervised training.

What Problem Does This Paper Address?

Automatic Question Generation (QG) is a crucial task in Natural Language Understanding (NLU), aiming to automatically create questions from given text. Traditionally, QG systems rely on large datasets of sentence-question pairs. While such datasets are abundant for languages like English, they are notoriously difficult and expensive to create for many other languages. This scarcity of training data poses a significant hurdle for building effective QG systems in low-resource languages. The paper addresses this gap by exploring how to effectively transfer knowledge from a high-resource language (English) to low-resource languages (Hindi and Chinese) for the QG task.

Methods

The core of the proposed solution is a Cross-Lingual QG (CLQG) model built upon a shared encoder-decoder architecture, specifically using the Transformer model. The training process involves two main phases:

  1. Unsupervised Pretraining: This phase leverages monolingual text from both the primary and secondary languages. It employs two techniques:
  2. Denoising Autoencoding: The model learns to reconstruct sentences from their corrupted (word-order permuted) versions. This helps the model learn grammar and vocabulary specific to each language while encouraging a shared latent space.
  3. Back-Translation: This technique generates pseudo-parallel corpora by translating sentences from one language to another and then back. This further strengthens the shared latent space assumption and is crucial for unsupervised NMT.
  4. Supervised Question Generation Training: After unsupervised pretraining, the model is fine-tuned using available sentence-question pairs. This phase involves joint supervised training for QG in both the primary and secondary languages, allowing the model to learn the specific QG task.

Key architectural choices include using positional masks to inject temporal information into the Transformer, and importantly, weight sharing between the encoder and decoder layers. Unlike some prior work, they don't share the entire encoder but rather specific layers, allowing for language-specific information capture while still benefiting from cross-lingual transfer. They also utilize BPE (Byte Pair Encoding) embeddings for more fine-grained control and a smaller vocabulary.

Datasets

The paper introduces a new dataset and utilizes several existing ones:

  • HiQuAD (Hindi Question Answering dataset): A newly created dataset for Hindi, containing 6555 question-answer pairs derived from 1334 paragraphs. This dataset is crucial for training and evaluating QG models for Hindi.
  • English SQuAD Dataset: A widely-used English question answering dataset, leveraged as the secondary language resource for QG training.
  • IITB Hindi Monolingual Corpus: Used for unsupervised pretraining of the Hindi language model.
  • IITB Parallel Corpus (English-Hindi): Used for further fine-tuning the model after unsupervised pretraining.
  • DuReader (Chinese Dataset): Used for QG training and as a Chinese monolingual corpus for unsupervised pretraining.
  • News Commentary Dataset (Chinese): A parallel corpus used for fine-tuning the Chinese model.

Key Insights and Ideas

  • Leveraging Scarce Data: The central idea is that even a small amount of QG data in a target language can be significantly boosted by leveraging a large QG dataset from a related or high-resource language.
  • Unsupervised Pretraining is Key: The unsupervised pretraining phase, using denoising autoencoding and back-translation, is critical for establishing a shared latent space and learning language-specific representations before supervised QG training. This is what enables effective cross-lingual transfer.
  • Weight Sharing Strategy: The paper explores a specific weight sharing strategy (sharing some encoder and decoder layers) that balances language-specific learning with cross-lingual knowledge transfer, proving effective in their experiments.
  • Importance of Parallel Data: While unsupervised pretraining is foundational, fine-tuning with parallel corpora (even if smaller or domain-specific) can further improve QG performance, suggesting a multi-stage transfer learning approach.

Key Results

The CLQG model, especially when fine-tuned with parallel data (CLQG+parallel), significantly outperforms baseline Transformer models on both Hindi and Chinese QG tasks.

  • Hindi: The CLQG+parallel model achieved the best results across various metrics, including BLEU, METEOR, and ROUGE-L. For instance, it achieved a BLEU-4 score of 20.242, compared to the baseline Transformer's 8.644.
  • Chinese: The CLQG model also showed improvements over the baseline Transformer, though the gains from parallel fine-tuning were less pronounced, potentially due to the smaller size and domain specificity of the Chinese parallel corpus.
  • Human Evaluation: A human evaluation study on Hindi demonstrated that the CLQG+parallel model performed significantly better than the baseline Transformer on semantic correctness and relevance, with comparable syntactic correctness.

Limitations and Future Work

  • Domain Specificity: The paper notes that parallel fine-tuning for Chinese yielded less improvement, possibly due to the domain-specific nature of the parallel corpus (news domain). This suggests that domain alignment between the source and target data can be important.
  • Cross-Lingual Embeddings: The authors suggest exploring the use of cross-lingual embeddings as a future direction to further enhance performance.
  • Data Requirements: While the paper demonstrates the effectiveness of cross-lingual transfer, it also implicitly highlights the trade-off between the size of the primary language QG dataset and performance. Figure 6 shows that even with cross-lingual pretraining, performance continues to improve with more primary language data.

Summary updated

Abstract

Automatic question generation (QG) is a challenging problem in natural language understanding. QG systems are typically built assuming access to a large number of training instances where each instance is a question and its corresponding answer. For a new language, such training instances are hard to obtain making the QG problem even more challenging. Using this as our motivation, we study the reuse of an available large QG dataset in a secondary language (e.g. English) to learn a QG model for a primary language (e.g. Hindi) of interest. For the primary language, we assume access to a large amount of monolingual text but only a small QG dataset. We propose a cross-lingual QG model which uses the following training regime: (i) Unsupervised pretraining of language models in both primary and secondary languages and (ii) joint supervised training for QG in both languages. We demonstrate the efficacy of our proposed approach using two different primary languages, Hindi and Chinese. We also create and release a new question answering dataset for Hindi consisting of 6555 sentences.
BibTeX
@inproceedings{kumar2019cross,
  author = {Vishwajeet Kumar and Nitish Joshi and Arijit Mukherjee and Ganesh Ramakrishnan and Preethi Jyothi},
  title = {{Cross-Lingual Training for Automatic Question Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=IZqjUSq_F1z},
}
Figure