Preserving Multilingual Quality While Tuning Query Encoder on English Only

Oleg Vasilyev, Randy Sawaya, John BohannonNAACL 2025 · DOI

Abstract

Fine-tuning a multilingual model on a specific English-only task typically risks "breaking" its performance in other languages, a phenomenon known as catastrophic forgetting. This paper introduces adiabatic tuning: a regime using intentionally low learning rates and frozen embedding layers. This approach not only preserves but often improves a model's multilingual and out-of-domain capabilities while optimizing for a narrow English target.

Introduction

In modern information retrieval, dual encoders represent queries and documents as dense vectors to enable rapid matching. When specializing a search system for a new domain (e.g., legal or medical), it is significantly more cost-effective to fine-tune only the query encoder rather than re-indexing millions of documents. However, researchers have long feared that fine-tuning a multilingual query encoder on a narrow, English-only dataset like MSMARCO would cause the model to lose its cross-lingual alignment and general text-processing capabilities.

Methods: Adiabatic Tuning

The authors propose adiabatic tuning, borrowing a term from thermodynamics to describe a process that occurs slowly enough for a system to remain in its equilibrium state. The strategy relies on three key technical components:

  1. Intentionally Low Learning Rates: Utilizing rates as low as $5 \times 10^{-8}$, which prevents the model from "jumping out" of the high-quality local minima established during pre-training.
  2. Partial Freezing: The document encoder is frozen entirely, and the embedding block of the query encoder is also frozen. This ensures the fundamental token "map" remains aligned across languages.
  3. Contrastive Learning: A standard triplet margin loss is used to pull relevant query-document pairs closer while pushing negatives away.

Experimental Results

The "Pull-Up" Effect

Surprisingly, tuning on English-only data actually improved performance on 14 other languages in the XNLI benchmark. This "pull-up" effect suggests that improving query representations in one language helps the model better navigate the shared multilingual vector space.

Heatmap showing performance improvements across various language pairs on the XNLI dataset after English-only tuning.

The Learning Rate Threshold

There is a critical "tipping point" in the tuning process. As shown in the evaluation across XNLI and English-only datasets (MSMARCO, arXiv), performance gains are stable at very low learning rates. However, once the learning rate exceeds approximately $1 \times 10^{-7}$ for the E5 model, the model begins to improve faster on the specific English task but rapidly loses its multilingual and out-of-domain robustness.

Graphs illustrating the relationship between learning rate and performance across multilingual (XNLI) and English-only datasets.

Out-of-Domain Robustness

The model demonstrated relative improvements in Positive-Negative Discrepancy (PND) on the arXiv dataset (up to ~5-7%) and SQuAD (~1%), despite these being stylistically distinct from the MSMARCO tuning data. This indicates that adiabatic tuning enhances the model's general retrieval capabilities rather than just memorizing the tuning distribution.

Conclusion

Adiabatic tuning provides a resource-light method (requiring roughly one hour on an A100 GPU) to specialize multilingual models. By maintaining embedding stability and using conservative learning rates, practitioners can adapt query encoders to specific domains without sacrificing the broad linguistic utility of the original model. Future work will explore whether this effect holds when tuning on low-resource languages to improve high-resource performance.

Summary updated

BibTeX
@inproceedings{vasilyev2025preserving,
  author = {Oleg Vasilyev and Randy Sawaya and John Bohannon},
  title = {{Preserving Multilingual Quality While Tuning Query Encoder on English Only}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-short.28},
}

Loading related content...

Figure