Can Cross-Lingual Transferability of Multilingual Transformers Be Activated Without End-Task Data?

Zewen Chi, Heyan Huang, Xian-Ling Mao

Abstract

Cross-lingual transfer typically requires fine-tuning multilingual models on labeled task data. However, data privacy often prevents access to the original training sets of high-performing monolingual models. This paper introduces PLUGIN-X, a modular framework that activates cross-lingual capabilities in existing monolingual task models without requiring the original training data. By disassembling models and reassembling them with a cross-model connector trained via Heterogeneous Masked Language Modeling (HMLM), PLUGIN-X achieves significant performance gains on cross-lingual benchmarks like XNLI and XQuAD compared to data-free baselines.

Introduction

In Natural Language Processing (NLP), cross-lingual transfer aims to adapt models trained in high-resource languages (e.g., English) to other languages. The standard paradigm involves fine-tuning a pretrained multilingual model (like mBERT or XLM-R) on labeled task data. However, a significant real-world constraint is the Public-Model-In-house-Data (PMID) setting: high-performing models are often released, but the proprietary datasets used to train them remain private due to legal or policy restrictions.

Without access to the original training data, traditional transfer methods fail. PLUGIN-X addresses this by asking whether cross-lingual potential can be "activated" using only model weights and unlabeled text in various languages.

Methodology: The PLUGIN-X Framework

The authors propose a "plug-and-play" approach that treats neural network architectures as modular components. The process is divided into three distinct phases: disassembly, reassembly, and adaptation.

Disassembly and Reassembly

First, a monolingual "End-Task Model" and a pretrained "Multilingual Model" are disassembled into their constituent sub-modules. These are then reassembled into a new pipeline:

  1. Multilingual Encoder: The lower layers of the multilingual model, responsible for general linguistic features.
  2. Cross-Model Connector: A set of new Transformer layers that bridge the representation gap between the encoder and the task head.
  3. End-Task Module: The upper layers and task head of the original monolingual model, which preserve the specific task "knowledge."
Overview of the PLUGIN-X framework showing disassembly of existing models and reassembly into a new model.

Representation Adaptation via HMLM

To ensure these disparate modules can communicate, the authors introduce Heterogeneous Masked Language Modeling (HMLM). This technique uses unlabeled text (e.g., from the CCNet corpus) to train the Connector to map the multilingual encoder's outputs into the representation space expected by the task module.

Illustration of the Heterogeneous Masked Language Modeling (HMLM) process.

Crucially, HMLM utilizes the monolingual model's vocabulary for the output predictions while using the multilingual model's vocabulary for the input tokens. This "heterogeneous" setup forces the Connector to learn the translation between the two internal "languages" of the models.

Experimental Setup and Results

The framework was evaluated on two major cross-lingual benchmarks: XNLI (Natural Language Inference across 15 languages) and XQuAD (Question Answering across 10 languages).

Performance Gains

PLUGIN-X demonstrated substantial improvements over previous data-free baselines:

  • XNLI: While simple embedding mapping (EMBMAP) achieved only 33.4% accuracy (near random for a 3-class task), PLUGIN-X reached 62.9%.
  • XQuAD: On question answering, PLUGIN-X achieved an F1 score of 47.0, whereas previous methods were largely non-functional, scoring between 1.1 and 7.0.

Representation Alignment

Analysis using $L_2$ distance and cosine similarity confirmed that the Connector effectively aligned the representations of different languages with the original English model's space. This supports the hypothesis that the primary barrier to zero-data transfer is representation mismatch rather than a lack of task-specific knowledge.

Discussion and Conclusion

PLUGIN-X leverages the "Universal Layer" hypothesis—the idea that middle layers of multilingual Transformers produce naturally aligned cross-lingual representations. By "plugging in" at these layers, the method achieves a form of model fusion that combines task expertise with linguistic breadth.

While PLUGIN-X currently lags behind "gold-standard" fine-tuning on private data, it establishes that cross-lingual transfer is viable even when training data is inaccessible. Future work may extend this modular approach to generative tasks and larger, more complex architectures like Large Language Models (LLMs).

Summary updated

Abstract

Pretrained multilingual Transformers have achieved great success in cross-lingual transfer learning. Current methods typically activate the cross-lingual transferability of multilingual Transformers by fine-tuning them on end-task data. However, the methods cannot perform cross-lingual transfer when end-task data are unavailable. In this work, we explore whether the cross-lingual transferability can be activated without end-task data. We propose a cross-lingual transfer method, named PlugIn-X. PlugIn-X disassembles monolingual and multilingual Transformers into sub-modules, and reassembles them to be the multilingual end-task model. After representation adaptation, PlugIn-X finally performs cross-lingual transfer in a plug-and-play style. Experimental results show that PlugIn-X successfully activates the cross-lingual transferability of multilingual Transformers without accessing end-task data. Moreover, we analyze how the cross-model representation alignment affects the cross-lingual transferability.
BibTeX
@inproceedings{chi2023can,
  author = {Zewen Chi and Heyan Huang and Xian-Ling Mao},
  title = {{Can Cross-Lingual Transferability of Multilingual Transformers Be Activated Without End-Task Data?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.796},
}
Figure