CL-ReLKT: Cross-lingual Language Knowledge Transfer for Multilingual Retrieval Question Answering

Peerat Limkonchotiwat, Wuttikorn Ponwitayarat, Can Udomcharoenchaikit, Ekapol Chuangsuwanich, Sarana NutanongNAACL 2022 · DOI

Introduction

Cross-Lingual Retrieval Question Answering (CL-ReQA) represents a critical frontier in natural language processing, enabling users to query information in one language and retrieve precise answers from documents written in another. However, the field is currently hampered by significant data imbalances and the high computational costs associated with large-scale transformer models. This paper introduces CL-ReLKT (Cross-lingual Retrieval Language Knowledge Transfer), a framework designed to bridge the performance gap between dominant and low-resource languages through efficient knowledge distillation.

The Challenge of Cross-Lingual QA

Developing robust CL-ReQA systems involves overcoming three primary hurdles:

  1. Language Imbalance: Most training corpora are heavily skewed toward English, leading to models that perform exceptionally well in dominant languages but fail in low-resource contexts.
  2. Fixed Language Support: State-of-the-art multilingual encoders, such as the Multilingual Universal Sentence Encoder (mUSE), often support only a limited set of languages. Performance typically degrades sharply for languages outside this predefined scope.
  3. Computational Overhead: Fine-tuning massive models like mBERT or XLM-R is resource-intensive and requires extensive parallel training data, which is often unavailable for many world languages.

Methodology: The CL-ReLKT Framework

The authors propose a two-stage "Teacher-Student" architecture to address these issues. By leveraging a lightweight CNN-based architecture, the framework achieves a 10x speedup in embedding computation compared to standard transformer-based approaches.

Stage 1: Teacher Model Preparation

The process begins by training a "Teacher" model using only English data. This model utilizes a triplet loss objective to structure the vector space, ensuring that questions and their corresponding answer documents are positioned closely together while pushing irrelevant documents away.

Stage 2: Knowledge Transfer

In the core innovation of this work, a "Student" model is initialized from the teacher's weights but trained to handle non-dominant languages. While the teacher's parameters remain frozen, the student is optimized using three distinct consistency objectives to align the multilingual vector space.

The CL-ReLKT training architecture, illustrating the three consistency objectives used to transfer knowledge from the frozen teacher model to the student model.
  • Obj 1 (Question-Question): Aligns the student’s encoding of a non-English question with the teacher’s encoding of its English translation.
  • Obj 2 (Document-Document): Ensures the student maintains the teacher's original accuracy in document encoding.
  • Obj 3 (Document-Question): Directly aligns non-English question embeddings with the teacher’s document embeddings to facilitate cross-lingual retrieval.

Conceptual Alignment and Insights

The primary strength of CL-ReLKT lies in its ability to use English as a "bridge." By mimicking the teacher's well-structured English vector space, the student model can generalize to languages it was never explicitly trained on. This "Power of the Pivot" allows for a more uniform distribution of knowledge across the language spectrum.

Visualization of the vector space before and after the knowledge transfer process, demonstrating improved alignment between questions and documents across languages.

Furthermore, the research highlights that retrieving at the document level—grouping related passages together—provides better context for downstream machine reading tasks than retrieving isolated snippets. This holistic approach helps the retriever identify the correct "neighborhood" in the vector space more reliably.

Experimental Evaluation and Results

The framework was evaluated across 15 languages using major benchmarks including XORQA, XQuAD, and MLQA. Key findings include:

  • Superior Accuracy: CL-ReLKT outperformed state-of-the-art competitors (including DPR and LaBSE) in 19 out of 21 experimental settings.
  • Unsupported Language Performance: In languages not originally supported by the base mUSE model, such as Finnish, CL-ReLKT improved Precision@1 from 18.3% to 48.2%.
  • Efficiency: The model achieved an average query encoding time of ~8.5ms, significantly faster than BERT-based models (~15-22ms) and machine-translation-assisted methods (~258ms).

Discussion and Future Directions

While CL-ReLKT offers a significant leap in efficiency and accuracy, limitations remain. The model occasionally struggles with numerical reasoning across languages and faces challenges with Indo-Aryan languages due to tokenizer-related "Out-of-Vocabulary" (OOV) issues. Future research will focus on integrating more robust tokenizers and improving the model's ability to map complex numerical concepts across disparate linguistic contexts.

Summary updated

BibTeX
@inproceedings{limkonchotiwat2022cl,
  author = {Peerat Limkonchotiwat and Wuttikorn Ponwitayarat and Can Udomcharoenchaikit and Ekapol Chuangsuwanich and Sarana Nutanong},
  title = {{CL-ReLKT: Cross-lingual Language Knowledge Transfer for Multilingual Retrieval Question Answering}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-naacl.165},
}
Figure