Efficient Test Time Adapter Ensembling for Low-resource Language Varieties

Xinyi Wang, Yulia Tsvetkov, Sebastian Ruder, Graham Neubig

Introduction

Massively multilingual models like mBERT provide a strong foundation for cross-lingual transfer, yet adapting them to the "long tail" of the world's 7,000+ languages remains a significant challenge. A common solution involves adapters—small, modular components added to a frozen model to handle specific languages or tasks. However, the current state-of-the-art framework, MAD-X, requires a dedicated adapter for every target language. This creates a "catch-22" for low-resource language varieties and dialects (e.g., Bhojpuri or Faroese): training an adapter requires data, but these varieties are often so under-resourced that even unlabeled text is scarce. Simply using an adapter from a related high-resource language (like using a Hindi adapter for Marathi) often leads to sub-optimal performance due to over-specialization.

Methods: Entropy Minimized Ensemble of Adapters (EMEA)

To address the data scarcity problem, the authors propose moving away from the "one language, one adapter" paradigm toward a collaborative approach.

Comparison between standard language/task adapters and the proposed EMEA architecture.

The core innovation is EMEA (Entropy Minimized Ensemble of Adapters). Instead of selecting a single "best" related adapter, EMEA takes a weighted average of outputs from multiple existing adapters. The ensemble weights ($\alpha$) are treated as learnable parameters at test time. For a given test sentence, the model calculates the entropy (uncertainty) of its predictions and performs a few steps of gradient descent to update the weights to minimize that entropy. This allows the model to "self-adapt" to each sentence by finding the combination of adapters that yields the highest confidence.

Experimental Setup

The researchers evaluated EMEA on two sequence labeling tasks: Named Entity Recognition (NER) using WikiAnn and Part-of-Speech (POS) tagging using Universal Treebank 2.0. They focused on three language clusters to simulate dialectal challenges:

  • Hindi Group: Hindi (base); Marathi, Bengali, Tamil, and Bhojpuri (uncovered).
  • Icelandic Group: Icelandic (base); Faroese, Norwegian, and Danish (uncovered).
  • Russian Group: Russian (base); Bulgarian, Ukrainian, and Belorussian (uncovered).

Key Results and Insights

EMEA demonstrates that prediction confidence is a robust proxy for accuracy in cross-lingual transfer. By forcing the model to be more certain, it naturally identifies linguistically relevant adapter combinations.

Performance comparison of EMEA versus training a new adapter with varying amounts of monolingual data for Marathi (mr).

As shown in the figure above, EMEA can achieve performance comparable to training a brand-new adapter with 100,000 lines of monolingual data, despite requiring zero target-language training data. In NER tasks, EMEA significantly improved F1 scores across the board; for the Hindi group, the average F1 rose from 44.0 (English baseline) to 53.0. Unlike other baselines such as "Fusion" or "Continual Learning," EMEA provided consistent improvements across different language families.

Limitations and Future Work

While effective, EMEA introduces a trade-off in inference speed. Performing gradient descent at test time is computationally expensive; EMEA-s10 (10 gradient steps) processes approximately 9 examples per second, compared to 250 for a single adapter. Additionally, the current study focused on sequence labeling. Future work should explore how entropy minimization scales to generative tasks with larger output spaces and investigate more efficient optimization strategies to reduce test-time overhead.

Summary updated

Abstract

Adapters are light-weight modules that allow parameter-efficient fine-tuning of pretrained models. Specialized language and task adapters have recently been proposed to facilitate cross-lingual transfer of multilingual pretrained models (Pfeiffer et al., 2020b). However, this approach requires training a separate language adapter for every language one wishes to support, which can be impractical for languages with limited data. An intuitive solution is to use a related language adapter for the new language variety, but we observe that this solution can lead to sub-optimal performance. In this paper, we aim to improve the robustness of language adapters to uncovered languages without training new adapters. We find that ensembling multiple existing language adapters makes the fine-tuned model significantly more robust to other language varieties not included in these adapters. Building upon this observation, we propose Entropy Minimized Ensemble of Adapters (EMEA), a method that optimizes the ensemble weights of the pretrained language adapters for each test sentence by minimizing the entropy of its predictions. Experiments on three diverse groups of language varieties show that our method leads to significant improvements on both named entity recognition and part-of-speech tagging across all languages.
BibTeX
@inproceedings{wang2021efficient,
  author = {Xinyi Wang and Yulia Tsvetkov and Sebastian Ruder and Graham Neubig},
  title = {{Efficient Test Time Adapter Ensembling for Low-resource Language Varieties}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.63},
}
Figure