McCrolin: Multi-consistency Cross-lingual Training for Retrieval Question Answering

Peerat Limkonchotiwat, Wuttikorn Ponwitayarat, Lalita Lowphansirikul, Potsawee Manakul, Can Udomcharoenchaikit, Ekapol Chuangsuwanich, Sarana NutanongEMNLP 2024 · DOI

Introduction

McCrolin is a novel multi-task training framework designed to enhance how AI systems retrieve answers across diverse languages. By optimizing for cross-lingual consistency, ranking stability, and robustness to varying input granularities, McCrolin achieves state-of-the-art performance on major benchmarks and demonstrates strong generalizability across different model architectures.

The Challenge of Cross-lingual Retrieval

In modern Question Answering (QA) systems, retrieval is the foundational step. A system must often find answers in documents written in languages different from the user's query—a task known as cross-lingual retrieval. Current approaches face three primary hurdles:

  1. Inconsistency: Models may fail to map identical concepts in different languages to the same vector space.
  2. Ranking Instability: Maintaining a clear distinction between correct and incorrect answers across languages is difficult.
  3. Input-Size Sensitivity: Models optimized for specific text lengths (e.g., paragraphs) often perform poorly when tasked with retrieving sentences or full documents.

The McCrolin Framework

McCrolin addresses these gaps through a unified multi-task learning approach based on a Teacher-Student knowledge transfer setup. A "frozen" teacher model (mUSE) provides multilingual embeddings, while a student model is trained using three reciprocal loss components.

Overview of the McCrolin architecture, illustrating the teacher-student knowledge transfer and the three primary loss components.
  • Cross-lingual Consistency Loss ($L_{XLC}$): Ensures that non-English queries and their English translations produce nearly identical embeddings, anchoring multilingual representations to a shared space.
  • Ranking Loss ($L_{rank}$): Uses contrastive learning to transfer high-quality English ranking logic to other languages, forcing the model to cluster positive query-answer pairs while separating negative ones.
  • Hierarchical Alignment Loss ($L_{HA}$): Aligns representations across different granularities, training the model to recognize that a paragraph and the query it answers should occupy similar spaces regardless of text length.

Experimental Evaluation

The framework was evaluated on three gold-standard cross-lingual QA benchmarks: XORQA (open-retrieval across 7 languages), XQuAD (span extraction in 11 languages), and MLQA (multi-way aligned extractive QA). To test generalizability, McCrolin was applied to various pre-trained language models, including mBERT, XLM-R, and mE5.

Key Findings and Performance

McCrolin establishes a new state-of-the-art (SOTA) in cross-lingual retrieval. On average, it achieved a Recall@1 of 64.2%, surpassing the previous best of 62.4% held by CL-ReLKT.

Comparative performance of McCrolin and baseline models on cross-lingual benchmarks, including ablation results for individual loss components.

The Power of Reciprocity

A key insight is that the three training objectives are mutually beneficial. Improving ranking stability enhances cross-lingual consistency, and vice-versa. By treating English as a universal anchor, the model leverages abundant English data to "lift" the performance of lower-resource languages.

Versatility and Efficiency

McCrolin is architecture-agnostic. When applied to the mE5 base model, it boosted average performance from 66.4 to 74.5 points. Furthermore, in downstream Machine Reading Comprehension tasks using GPT-3.5, McCrolin's superior retrieval enabled higher F1 scores with fewer tokens, suggesting potential cost savings in real-world deployments.

Future Directions

Despite its success, McCrolin faces limitations. It struggled with languages like Hindi (HI) due to tokenizer constraints in the base models, highlighting a dependency on the underlying vocabulary. Future work will explore performance in specialized domains (e.g., legal or medical) and evaluate robustness against noisy, "in-the-wild" data from web crawls or chat logs.

Summary updated

BibTeX
@inproceedings{limkonchotiwat2024mccrolin,
  author = {Peerat Limkonchotiwat and Wuttikorn Ponwitayarat and Lalita Lowphansirikul and Potsawee Manakul and Can Udomcharoenchaikit and Ekapol Chuangsuwanich and Sarana Nutanong},
  title = {{McCrolin: Multi-consistency Cross-lingual Training for Retrieval Question Answering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.157},
}
Figure