Cross-Lingual Training for Automatic Question Generation

Vishwajeet Kumar, Nitish Joshi, Arijit Mukherjee, Ganesh Ramakrishnan, Preethi Jyothi

Introduction

Automatic Question Generation (QG) is a critical task for developing intelligent tutoring systems and enhancing search engine capabilities. While neural models have achieved high proficiency in English, they typically require vast amounts of labeled data—often hundreds of thousands of sentence-question pairs. For most other languages, such datasets are scarce, creating a significant barrier to entry.

This research addresses this "cold-start" problem by introducing a cross-lingual framework that allows models to leverage knowledge from high-resource languages (like English) to improve performance in low-resource settings (like Hindi and Chinese). Alongside this framework, the authors introduce HiQuAD, a new benchmark dataset for Hindi question answering.

Methodology

The proposed Cross-Lingual Question Generation (CLQG) model is built upon the Transformer architecture and utilizes a two-phase training regime designed to maximize cross-lingual transfer.

Training Phases

  1. Unsupervised Pre-training: The model first learns the linguistic structure of both the primary (e.g., Hindi) and secondary (English) languages using large-scale unpaired text. This phase employs Denoising Autoencoding (reconstructing shuffled sentences) and Back-translation (translating from English to Hindi and back to English). These techniques force the model to develop a shared latent space where semantic meaning is preserved across languages.
  2. Joint Supervised Training: The model is fine-tuned simultaneously on actual sentence-question pairs from both languages.

Architectural Innovation

Unlike standard models that share all parameters, CLQG uses "partial weight sharing." Language-specific encoder and decoder layers are maintained to capture unique grammatical nuances, while the middle layers are shared to facilitate the transfer of abstract semantic concepts.

The CLQG architecture showing shared and language-specific weights across different training phases.

Datasets

The study utilizes several key datasets to validate the model's effectiveness:

  • HiQuAD: A novel Hindi dataset containing 6,555 sentence-question pairs curated from Indian history texts.
  • SQuAD: The standard English dataset (approx. 70,000 pairs) serving as the high-resource secondary language.
  • DuReader: A Chinese dataset used to evaluate generalizability across different language families.
  • Monolingual Corpora: Large-scale unpaired text for Hindi (93,000 sentences) and Chinese (100,000 descriptions) used during unsupervised pre-training.

Results and Discussion

The CLQG model demonstrated substantial improvements over baseline architectures in both quantitative and qualitative evaluations.

Quantitative Performance

For Hindi, the CLQG+parallel model achieved a BLEU-4 score of 20.24, more than doubling the performance of a standard Transformer baseline (8.64). Similar, though less dramatic, gains were observed in the Chinese experiments.

Comparison of BLEU, METEOR, and ROUGE-L scores for Hindi and Chinese across various model configurations.

Key Insights

  • The Impact of Pre-training: Ablation studies identified unsupervised pre-training as the most significant contributor to performance, providing a jump of approximately 7 BLEU-4 points. This phase effectively teaches the model grammar before it begins learning the specific task of question formulation.
  • Cross-Lingual Logic: Joint training on English data improved Hindi performance, suggesting the model successfully learns the abstract logic of question generation and applies it across languages.
  • Data Efficiency: The model showed high efficiency, achieving significant performance gains even when the Hindi training set was restricted to only 2,000 examples.

Conclusion and Future Work

The CLQG framework proves that cross-lingual transfer and unsupervised pre-training can effectively mitigate data scarcity in QG tasks. While domain mismatches between datasets (e.g., Wikipedia vs. history books) remain a challenge, the results establish a strong baseline for low-resource QG. Future research will explore the integration of cross-lingual embeddings, such as XLM, to further refine language alignment during pre-training.

Summary updated

Abstract

Automatic question generation (QG) is a challenging problem in natural language understanding. QG systems are typically built assuming access to a large number of training instances where each instance is a question and its corresponding answer. For a new language, such training instances are hard to obtain making the QG problem even more challenging. Using this as our motivation, we study the reuse of an available large QG dataset in a secondary language (e.g. English) to learn a QG model for a primary language (e.g. Hindi) of interest. For the primary language, we assume access to a large amount of monolingual text but only a small QG dataset. We propose a cross-lingual QG model which uses the following training regime: (i) Unsupervised pretraining of language models in both primary and secondary languages and (ii) joint supervised training for QG in both languages. We demonstrate the efficacy of our proposed approach using two different primary languages, Hindi and Chinese. We also create and release a new question answering dataset for Hindi consisting of 6555 sentences.
BibTeX
@inproceedings{kumar2019cross,
  author = {Vishwajeet Kumar and Nitish Joshi and Arijit Mukherjee and Ganesh Ramakrishnan and Preethi Jyothi},
  title = {{Cross-Lingual Training for Automatic Question Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=IZqjUSq_F1z},
}
Figure