X-METRA-ADA: Cross-lingual Meta-Transfer Learning Adaptation to Natural Language Understanding and Question Answering

Meryem M'hamdi, Doo Soon Kim, Franck Dernoncourt, Trung Bui, Xiang Ren, Jonathan May

Introduction

Multilingual pre-trained models like M-BERT and XLM-R have revolutionized cross-lingual transfer. However, their zero-shot capabilities often falter when applied to typologically diverse languages—those with grammatical structures significantly different from the source language (usually English)—or in extreme low-resource scenarios. Standard fine-tuning on high-resource data often leads to overfitting on source-specific features, hindering generalization to distant target languages.

The X-METRA-ADA Framework

To address these challenges, we introduce X-METRA-ADA, an optimization-based meta-learning approach built upon Model-Agnostic Meta-Learning (MAML). Instead of direct weight optimization for a specific task, X-METRA-ADA identifies a model initialization point that is maximally adaptable across language boundaries.

The X-METRA-ADA framework: A two-stage process involving cross-lingual meta-training and language-specific meta-adaptation.

The framework operates in two distinct stages:

  1. Meta-train: The model learns to transfer knowledge from a high-resource source (English) to a target language. In this phase, the "support set" (inner loop) uses English data, while the "query set" (outer loop) uses target language data.
  2. Meta-adapt: The model undergoes further refinement using only limited target language data for both support and query sets, capturing language-specific nuances.

For Question Answering (QA), which lacks the discrete classes of intent classification, we utilize cross-lingual SBERT embeddings to group semantically similar question-context-answer triplets into cohesive "pseudo-tasks," enabling the application of MAML to span-extraction tasks.

Experimental Setup

We evaluated X-METRA-ADA on two primary benchmarks:

  • MTOD (Multilingual Task-Oriented Dialogue): Focused on joint intent classification and slot filling across English, Spanish, and Thai.
  • TyDiQA-GoldP: A typologically diverse QA dataset covering seven languages, including Arabic, Bengali, Finnish, Indonesian, Russian, Swahili, and Telugu.

Key Results and Insights

Our experiments demonstrate that treating cross-lingual transfer as a meta-learning problem provides a more stable and generalizable starting point than standard fine-tuning.

Intent accuracy scores over training steps for Spanish (ES) and Thai (TH), comparing X-METRA-ADA against standard fine-tuning (FT).

A critical finding is that the benefits of meta-learning are most pronounced for languages typologically distant from English. For instance, in Thai (MTOD), X-METRA-ADA achieved 86.3% intent accuracy, significantly outperforming the 78.9% achieved by standard fine-tuning. In contrast, for typologically similar languages like Spanish, the performance gap is narrower, and standard fine-tuning remains highly competitive.

In the TyDiQA benchmark, the approach yielded an average increase of 3.2% over the fine-tuning baseline. Notable gains were observed in Bengali (57.8 F1 vs. 51.0 F1) and Telugu (72.8 F1 vs. 65.4 F1). Furthermore, downsampling analysis confirms that X-METRA-ADA maintains its edge even when target language data is extremely scarce, demonstrating high data efficiency.

Conclusion and Future Work

X-METRA-ADA effectively bridges the gap for low-resource, typologically diverse languages in NLU tasks. While meta-learning offers substantial gains for distant languages, its added complexity may be unnecessary for closely related language pairs where standard fine-tuning suffices. Future research will explore the direct statistical correlations between specific linguistic features and performance gains, alongside deeper investigations into the role of linear layers in cross-lingual alignment.

Summary updated

BibTeX
@inproceedings{mhamdi2021x,
  author = {Meryem M'hamdi and Doo Soon Kim and Franck Dernoncourt and Trung Bui and Xiang Ren and Jonathan May},
  title = {{X-METRA-ADA: Cross-lingual Meta-Transfer Learning Adaptation to Natural Language Understanding and Question Answering}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.283},
}
Figure